Commercial AID or Open Source? Some things to consider…

This is a classic question that comes up frequently within groups that are largely populated by members of the ¥WeAreNotWaiting community. It’s often included with statements such as “Why would you use a commercial system?” or “But people get so much better results with Open Source”.

The key issue is that there are both differences and similarities between the open source systems and commercial ones. There are also similarities in outcomes which tend to get overlooked. In addition, there are plenty of confounding factors that can make anecdotal evidence unreliable when asking a question like this.

This is a brief essay that raises some of those considerations, looks at the questions that should be asked when anyone says how great their system is and opens it up for debate.

So let’s make a start:

Commercial systems are designed to be used by absolutely anyone.

This means that a user doesn’t have to understand the minutiae of setting up profiles and adapting them on a regular basis to cope with the changes that life brings. With this, there is also the down side that targets are generally higher, and there is less flexibility about setup, but outside the #WeAreNotWaiting bubble (which really is very small) these are a godsend.

The general T1D population is seeing results that have only been available to those using Open Source, without the need to build anything and with far fewer requirements to set things up. Just enter your weight, age and TDD and off you go.

Settings and set-up

Open source systems demand that they are set up reasonably well. They are mostly based off common diabetes maths and as a result, need a modicum of careful configuration to get the best out of them. But they also come with a limited amount of settings automation which means that those settings need to be adapted to get best results over time. Users often spend a lot of time making those adjustments.

As an adjunct of of this, open source systems make a wide variety of settings available to the user to twiddle with. In many cases you probably don’t need to but humans are not very good at taking a step back when there are knobs to twiddle, and instead of understanding what’s happening, will tweak things, which may or may not work.

You might argue that this is an issue with OS AID. They would perhaps work well without exposing so much to the user, who would be better slated to use observation and adaptation to change basic settings rather than tweak everything.

The user community itself

The #WeAreNotWaiting community is widely recognised as being more engaged with their care or with that of their loved ones than the wider type 1 population.

All may not be what it seems 

People publicly showing high time in range and time in tight range may not just be using an AID system. There are numerous factors that may be involved with outcomes. This list is not exhaustive, however, it highlights that it’s always worth asking further questions when presented with data.

The first four are applicable to any AID system, open source or commercial. The final one is more a nuance of how open source systems work.

• Adjunct therapies – GLP1 Receptor Agonists; SGLT2 inhibitors; Metformin. Any of these drugs change the way the body deals with food, and therefore how an AID manages it. I’d argue that #WeAreNotWaiting users are more likely to be at the forefront of trying adjunct therapies out, so it’s always worth asking or explaining what you are using.
• Food/lifestyle – do you know what someone eats? Are they very low carb, high fat or very low fat, high (raw) carb? Food choices can and do have a major impact on glucose levels outside of use of an AID system, so it stands to reason that they would affect outcomes with use of an AID system.
• Manual intervention – one thing that anecdotal response doesn’t account for is how much a user “takes over”. Without knowing how much someone does manually on top of an AID system, it’s impossible to get a feel for the capabilities of a system. The “Pig Experiment” highlights this.
• Selection Bias – very simply, those who publish numbers in community groups tend to be the ones with higher time in range values. Those who don’t tend to be the ones asking for help. More on this later.
• Vanilla Algorithms – this is more geared towards the #WeAreNotWaiting community. It’s a question of what the user is actually doing. Within AAPS, for example, it’s possible to easily create an “over-algorithm” that creates a rule based structure that changes how oref1 works. It’s also possible to use different branches of what looks like a standard algorithm. If you don’t know what modifications someone has made you can’t know how well the outcomes they display reflect the core algorithm.

What does this mean?

Anecdotal feedback about time in range from users of any system, without clear statement of how they get there makes comparing results quite hard. Two users of the same algorithm may have widely varying results by virtue of how well they count carbs. Whose experience is valid?

Essentially, while there’s often not a lot of fondness for outcomes from trials, they do normalise outcomes from all sorts of systems and give a much better idea of what you can expect from using any kind of system. And that’s why CREATE matters.

Community Derived Automated Insulin Delivery (CREATE)

CREATE was a trial study of “non-#WeAreNotWaiting” users on the oref1 algorithm in a modified version of AndroidAPS. It showed a mean TIR (70-180) for adults of 74.5% and for children of 67.5%. it’s a large enough population to allow comparison with some commercial trials. Some of those can be found here. The numbers from CREATE are very similar to those of other systems. The difference is certainly not statistically significant. It showed that oref1 based systems can compete with commercial systems.

It’s worth noting this text from the CREATE method:

All the patients attended three in-person visits (at weeks 0, 12, and 24); those in the AID group also had two additional reviews by telephone at weeks 3 and 6. During visits and telephone reviews, patients were asked about adverse events and device issues, medication use, and Dexcom alarm settings. In addition, staff members reviewed data using a unique URL provided by Nightscout (an open-source project that enables access to data regarding continuous glucose monitoring) and advised patients on changes in device settings. All the patients (or their parents or guardians) could alter settings between contacts, but staff members were instructed to avoid surveillance outside of scheduled reviews and did not receive automated alerts. This approach was designed to negate an effect on outcomes caused by additional scheduled contact with the trial team in the AID group.

This highlights that in use in the trial, there was ongoing advice at regular intervals to ensure that users were getting the best from the system, similar to the model the Tidepool have proposed for Tidepool Loop. Something that might exist in the real world in commercial systems, but doesn’t really exist in open source, where the reliance is on the user to check and ask questions.

This is important is because it allows us to recognise where to level set. We should understand that in a more general population of users with T1D, open source AID systems are no less magic than commercial ones. This comes back to the Selection Bias bullet on the list earlier. If, in a general population, the mean TIR is around 70%-75%, then even in an engaged population there are likely to be users who aren’t getting to those numbers. They just don’t talk about it.

Where does this leave us?

There’s a lot more to choosing an AID system than “Is it open source?”. As we’ve heard in various Loop and Learn sessions, parents with older kids are finding that leaving the open source world is a way to allow their kids to be released from continuous oversight, which has nothing to do with any of the above.

But it also demonstrates why personal choice and full disclosure from existing users is important. The world doesn’t always want a complex solution that can be challenging to set up and maintain. For many, something simpler with (generally) equivalent results will do and there are often multiple ways to get those.

The #WeAreNotWaiting community is the exception rather than the rule, and we should remember that when discussing outcomes and results in Automated Insulin Delivery use.