Dynamically change array length

[0UserName]

Out of curiosity.

internal readonly ref struct UnsafeResizeScope(object[] array) : IDisposable
{
    /// <summary>
    /// See: <see href="https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs#L297">RuntimeHelpers.RawArrayData</see>
    /// </summary>
    private class RawArrayData
    {
        public uint Length; // Array._numComponents padded to IntPtr
        //#if TARGET_64BIT
        public uint Padding;
        //#endif
        public byte Data;
    }

    private readonly int _length = array.Length;

    public void Resize(int length)
    {
        Unsafe.As<RawArrayData>(array).Length = (uint)length;
    }

    /// <inheritdoc/>
    public void Dispose()
    {
        Resize(_length);
    }

    public UnsafeResizeScope(object[] array, int length) : this(array)
    {
        Resize(length);
    }
}

This is a wrapper that allows dynamic resizing of an array. For example, we have an array, change its length, perform some operation on it (such as copying data elsewhere), and then restore its original length.

While this approach appears to work, after approximately 100000 iterations, an ExecutionEngineException is thrown. Why does this happen?

I suspect the issue is related to the GC, but it’s still unclear exactly how.

[tannergooding]

Because you've corrupted GC data and potentially made memory accessible that is not meant to be accessible. Possibly overwriting other data, particularly if the GC decides to copy or move the now incorrectly sized array.

Do unsafe things, bad things can happen.

[0UserName]

GC decides to copy or move the now incorrectly sized array

Is there a way to avoid this? By specifying GC to not touch the array for the duration of the operation.

and potentially made memory accessible that is not meant to be accessible

Do you mean data beyond the set length? Are memory areas simply becoming writable and causing access collisions?

[tannergooding]

Is there a way to avoid this? By specifying GC to not touch the array for the duration of the operation.

No. This is fundamentally unsafe, there is no way to make it safe. You are completely in undefined behavior territory when you resize an array, because now you're improperly tracking the backing allocation.

In the case you undersize the array (tracking less than was actually allocated), then anything looking at that will be missing data. If the GC decides to copy it, then you will now be missing data and have corrupted the copy. Additionally, this should've just been handled with span and creating a sliced view over the data, so it is safe.

In the case you oversize the array (tracking more than was actually allocate), then you risk things touching memory they shouldn't. This could be the GC copying the data, it could be something else reading an element it now thinks is valid. Such memory could be illegal to access (page isn't mapped), it could belong to another object, etc

This is fundamentally incorrect code, it should never be written.

Do you mean data beyond the set length?

Yes.

Are memory areas simply becoming writable and causing access collisions?

All memory is always possible to "attempt" to write. Such memory may actually be writeable which leads to corruption or may be readonly/unmapped and lead to some kind of access fault.

[huoyaoyuan]

A closer explanation why it breaks:

Objects are stored compactly on GC heap. GC scans objects one by one on the heap, and computes the location of next object simply by pPrevious + pPrevious->Length + objectHeaderSize. If you modify the length field, GC will loose the track of next object, and crash when attempting to manipulate subsequent object.

In the luckiest case, the length can happen to cover the next object exactly and direct GC to the second next object. This is totally unreliable since GC can place objects in arbitrary order onto the heap.

[tannergooding]

This is also an implementation detail of the current CoreCLR GC implementation.

It is not a guarantee, it is subject to change at any time, and may differ for other runtimes or GC implementations (Mono, Unity, etc).

Every bit of the code in the original example is relying on implementation detail and undefined behavior, it is simply code that users should never write.

[jkotas]

Unsafe.As<RawArrayData>(array).Length = (uint)length; will always produce intermittent crashes, data corruptions and hangs on current CoreCLR. There is no way to make it reliable.

I understand that having ability to resize an array would be convenient. You can find number of discussions about that in this repo. Unfortunately, there is no way to implement a reliable API like that without severe performance and security tradeoffs. A reliable API like that would be either very slow and/or we would have to add overhead to perf critical parts of the runtime to make that API reliable.

[Clockwork-Muse]

As a side note; virtually no computer language allows you to resize an array, because similarly to the GC there are practical realities for tracking available memory, and leaving space "off the end" would most often be wasting it.