Is the experiment checklist customizable? Can we add additional steps that may be specific to our project or teams?

Adding `Confirmation page shown` as the last metric in `test funnel metric`

Yeah agreed. I can’t come up with a timeline right now because the “aggregated stats for diagnostics” is something that I just thought about today in our conversation, so I will need to bring it to our team to prioritize among all the other things the team is building, but it is not on our immediate roadmap.

Ultimately the questions I want to answer are: • what percentage of our users are failing to initialize statsig? • who are those users? My inclination was that it would be easier/faster for us to get that data by logging it ourselves, but if we’re the only customer asking for this and it’s not already on your roadmap, very fair of you to push back. Based on your response, if you’re already planning on building additional capabilities into the Statsig product, that might work for us as well. I don’t think the log stream will get us these answers, since that only shows very recent exposures, and doesn’t let us drill back into the past or get aggregated stats. But “features to show aggregated stats on things like you mentioned, how often a user is using fresh value vs. cached and etc” sounds very much like this would answer our questions. Do you have a rough sense of the timeline for these features? Historically the Statsig team has impressed me with how fast feature requests have been built :slightly_smiling_face: but even just a sense of whether this is on your immediate roadmap or longer-term roadmap would help me determine whether we want to wait for these stats to be built, or if it’s worth investigating an alternative (like forking the SDK) in the meantime.

In general providing more debugging info like this is a direction we are going with the product, e.g. we are surfacing evaluation reason (cache, localoverrides, etc) in the log stream with the new SDK you are now using, which wasn’t available in the old version, with it there is also “evaluation time”, which indicates how fresh the value is (the timestamp will be old if it’s using cache, and it’s available in the details page of log stream, which we will also make available in the users tab soon). I think going forward we will implement features to show aggregated stats on things like you mentioned, how often a user is using fresh value vs. cached and etc. I don’t think it’s realistic for us to commit to providing what you are asking for in the SDK anytime soon, so if that’s the route you want to try quickly now, I think it’s actually easier for you to fork the SDK repo and make whatever change you want to try there in the forked version. The only thing I will note is that you will probably want to change these https://github.com/statsig-io/react-sdk/blob/bff44dae8c202748267b6179af1f2fbfbb71597d/src/StatsigProvider.tsx#L142-L143|metadata in your forked version, so that it will be easier for us to debug if something’s wrong.

Thanks

Braze doesn't have a native integration with us, but we have several ways to export variant assignment (webhook, CDB integrations like Segment, Rudderstack, mParticle etc, and an API to export). Is there a CDB you currently use?

Thinking about this some more, I’d be flexible based on whatever is easiest to implement in the SDK, but my ideal might be a way to tell whether initialize timed out or succeeded every time we created a `StatsigProvider`, so we could more easily track the denominator here of all times Statsig is loaded. (If we only get failures, we’d have to approximate the denominator using other metrics.)

No problem! That's definitely something we want to do in Events Explorer - we just haven't prioritized working on it yet. We are very open to suggestions/feature requests here as we figure out what to build so let us know what you want! And yes, being better than Scuba and ODS combined is the goal :slightly_smiling_face:

This is helpful, thank you! Will Statsig be looking into supporting more ad hoc transformations on the Event Explorer? This would make Events Explorer more like a superset of Scuba + ODS in facebook terms :joy:

One thing that I think would be helpful for us is if we’re able to log on our end how often this issue is occurring. I don’t think we can do so with the current Statsig SDK. • My first thought was to add a gate that’s supposed to always pass for everyone and log in our app whenever this gate resolves to false/fail. This will catch whenever a new client is unable to call initialize, but won’t detect when a client uses stale/cached data from days or weeks prior because it successfully called initialize, but is unable to do so now. • Looking through `StatsigClient.ts` in source code, it seems to me like there is no way to tell if `fetchValues` resolves due to the initialize succeeding, or the timeout. • It does look like `save` in `StatsigStore.ts` saves the current timestamp into the cache, so it’s possible we could try some sort of logging based on this timestamp being too old? But there seem to be cases where it’s intentionally old. To my understanding, if a user doesn’t refresh the page, we will not initialize again, and they will (intentionally) keep using the gate values they were using when they first loaded the page. Would it be possible for you to somehow augment the SDK to give us an API we could use to log on our end when initialize fails? It would be sufficient for us to log if and only if the original call plus all 3 retries all fail.

You cannot currently do this automatically in events explorer (you would need to do two separate queries, one for B, one for A, and then calculate the ratio). However, you can do this with custom ratio metrics! Here are the docs, let me know if you have any questions: https://docs.statsig.com/metrics/create#4-ratio-metrics

Question: Reading the documentation of the Events Explorer, I was wondering what the best way is to calculate the ratio of 2 events? For example, I have event A for some process being started and event B for the process succeeding. How can I best monitor B/A?

sweet, thanks

Thanks for reporting - we just caught this earlier this morning and are pushing a fix, you can use https://staging.console.statsig.com for now since the fix is there, or go to the regular console URL and the fix should be there in an hour or so

Error when trying to add more tests to the “Group” section on an Experiment that is not started yet

hi, when saving updates to an experiment, getting this error

I think Juan Diego De Miguel Mateo might have done this or something similar. But what do you mean exactly? There is a js bundle hosted on this cdn which you can use: https://www.jsdelivr.com/package/npm/statsig-js

Created https://github.com/statsig-io/python-sdk/issues/8 to track

did you have any advance on this one Eli Spitzer?

Maggie (Statsig) are you able to help with this one?

Thanks Maggie

Not necessarily. When you log the event for Add Payment Details, you can include a metadata field that's `hours since account creation`, for example. Then create a custom metric that's the user count of Add Payment Details with the filter `hours since account creation < 24` Another option is to create a ratio metric `Add Payment Details / Create Account`. This will only count users that have both events in the same calendar day. So not exactly 24 hours.

So I would need to log two events: “Create Account” and “AddPaymentDetails” with timestamp?

When you log the event for Add Payment Details, can you include time since account creation in the metadata? If so, then you can create a filtered user count metric

*Question:* Hi there. I’m working on an experiment where I want to measure how many users complete an action (Add Payment Details) within 24 hours after creating an account. What would be the recommended setup and custom events?

Myroslav Sypa ^

anu (statsig) FYI one of our team members tried it and got “*Response Status: Failed to fetch (CORS or Network Issue)*“, but it worked when he changed the url from ```curl --request GET 'https://api.statsig.net/console/v1/experiments' --header 'STATSIG-API-KEY: console-xxxxXXXXxxxxXXXXxxxx' ``` to ```curl --request GET 'https://api.statsig.com/console/v1/experiments' --header 'STATSIG-API-KEY: console-xxxxXXXXxxxxXXXXxxxx'```

(just for tracking metrics)

Hi Is it possible to implement the client SDK within Google Tag Manager?

No problem!

Sure. I can do that. Thanks!

I see. There isn’t any other downside to call `shutdown` when it’s not initialized other than the exception getting raised. We should probably remove that at some point in the SDK. For now can you catch the error around `shutdown` ?

The problem is if, something went wrong in the initialization stage, then both the init stage and the desconstruct stage will raise exceptions. The latter one will complains about calling shutdown failed since statsig hasn’t been initialized.

Yes. The context is that I initialize statsig once the Spring bean container is up and just call shutdown once when the bean container is down.

Hi Roger Lin - usually you don’t need to check for that when you call `shutdown`, and it’s only recommended to call this once per app session when it’s about to exit. Can you share some context on why you need to check if it’s been initialized? And are you looking to call `shutdown` multiple times in the app’s lifecycle?

Hi Xiaoyu Yin - if the gate does not exist (typo for example), we think the best thing to do is to return `false` , so that your app continues to function correctly. It’s recommended that during development, you check the “log stream” under the diagnostics section for the gate you are using. If it’s set up correctly, you should see logs coming in near real time to verify that everything is correct.

Hello folks, I am currently using the Java SDK for integrating STATSIG into my Spring Boot application. I am wondering is there a way for me to check whether statsig has been initialized before calling the shutdown method in a ContextClosedEvent listener? Thanks for any help in advance.