Did you check MiMo correctly performed this unfamiliar task before posting this comment?

I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.

This was the main change for bzip2:

  @@ -33,19 +34,16 @@ def candidate_lengths(
       level: int = 9,
       pool: ThreadPoolExecutor | None = None,
   ) -> list[int]:
  -    """Compressed length of ``context + seq`` for each seq, sharing the context.
  +    """Compressed length of ``context + seq`` for each seq.
  
  -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
  -    state per candidate and feeds only that candidate. Identical to
  -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
  -    match search over ``context`` happens a single time.
  +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
  +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
  +    whole block, so every candidate recompresses the full context. Threads
  +    still scale because ``bz2`` releases the GIL.
       """
  -    base = zlib.compressobj(level)
  -    head = len(base.compress(context))
  
       def length_for(seq: bytes) -> int:
  -        clone = base.copy()
  -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
  +        return len(bz2.compress(context + seq, level))
  
       if pool is not None:
           return list(pool.map(length_for, sequences))

Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?

(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)

This difference doesn't matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don't feed it enough data to complete a block, it waits until you feed it more or finish the stream.

We can test it by going back to zlib:

       def length_for(seq: bytes) -> int:
  -        return len(bz2.compress(context + seq, level))
  +        return len(zlib.compress(context + seq, level))

At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream:

  MENENIUS:
  'Though all at once cannq

  MARCIUS:
  I'll fight
  'Though all at once cannq

  MARCIUannq
  
  MARCIUS:
  I'll fight
  'Though
  
  AUFIDIUS:
  If I fly, Marci
  
  AUFIDIUS:
  If I fly, Marci
  
  AUFID
  
  AUFIDIUS:
  If
  If I fly

I also tried LZMA for good measure:

       def length_for(seq: bytes) -> int:
  -        return len(bz2.compress(context + seq, level))
  +        return len(lzma.compress(context + seq))

The sample at temperature zero:

  MENENIUS:
  'Th
  
  A carbuncle enti
  
  , as big as thou
  
  
  A aa

This is followed by a lot of whitespace.

python-lz4 gives you all newlines after the prompt. I tried debugging it, and the compressed length of different candidate seqs is the same.

I have no idea what this means but I love it!

I’m currently reverse engineering a large enterprise app and the feature flag bloat is truly astounding. Imho excessive feature flag complexity is a symptom of management who are indecisive and mistrusted by the developers.

You need to be actively deleting them after the feature has gone live

i could see it getting complex with a lot of flags, inevitable you'll run into a situation where more than one flag is combined.

something like enabledFeature = (flag1 || flag2 || flag3) && flag4

then, down the road, you remove flag4. Hopefully the above would result in a compile error but you may be in a language or situation where a missing flag4 is interpreted as boolean false. That could cause all kinds of havoc in logical combinations like that are scattered all over the codebase. Even worse would be REST APIs retrieving flag values because who knows what's happening in that service code? Plus, you'd never know there was an issue until users start reporting missing or extra features showing up unless you have e2e tests for every possible combination of feature flags...

Surely the commenter means deleting them in the code, not just the management software. Otherwise, what's the point?

You need to find every occurrence of flag4 and remove references to it. So that boolean expression would need to have `&& flag4` removed.

What I’m saying is there may be references you don’t have access to.

Create story to add feature flag controlling access to the feature.

Immediately create story to remove said feature flag controlling access to the feature and review it during backlog refinements.

> Imho excessive feature flag complexity is a symptom of management who are indecisive and mistrusted by the developers.

In my experience, engineers aren't using them to account for managerial dithering, they're doing it for safe deployments and experiments and rollouts and such. A product with millions of users can easily have a tens or even hundreds of active switches at any moment (I'm assuming a large engineering team behind said product), and that's not necessarily a bad thing.

However, as someone else noted here, you absolutely MUST delete and clean up your flags/gates/whatever when you've completed that effort. That part can be tricky because not everyone has the discipline to pay off tech debt.

Usually, a flag/gate should not live in code for more than a few months. If it does, it should have robust justification.

This of course depends on the quantity and the depth of these feature flags. It’s ok to hard code few of them, maybe it’s not when it becomes a practice and you have tens of them. I think you would have a much easier life if there was a standardized, well documented, way to define them.