VirtualizingStackPanel with Virtualized List

Question

I am developing an application that is supposed to display a fairly large amount of items that are loaded from elsewhere (say, a database) in a list/grid-like thing.

As having all the items in memory all the time seems like a waste, I am looking into ways to virtualize a part of my list. VirtualizingStackPanel seems just like what I need - however, while it seems to do a good job virtualizing the UI of items, I am not sure how to virtualize parts of the underlying item list itself.

As a small sample, consider a WPF application with this as its main window:

<Window x:Class="VSPTest.Window1"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="VSPTest" Height="300" Width="300">
    <Window.Resources>
        <DataTemplate x:Key="itemTpl">
            <Border BorderBrush="Blue" BorderThickness="2" CornerRadius="5" Margin="2" Padding="4" Background="Chocolate">
                <Border BorderBrush="Red" BorderThickness="1" CornerRadius="4" Padding="3" Background="Yellow">
                    <TextBlock Text="{Binding Index}"/>
                </Border>
            </Border>
        </DataTemplate>
    </Window.Resources>
    <Border Padding="5">
        <ListBox VirtualizingStackPanel.IsVirtualizing="True" ItemsSource="{Binding .}" ItemTemplate="{StaticResource itemTpl}" VirtualizingStackPanel.CleanUpVirtualizedItem="ListBox_CleanUpVirtualizedItem">
            <ListBox.ItemContainerStyle>
                <Style TargetType="{x:Type ListBoxItem}">
                    <Setter Property="HorizontalContentAlignment" Value="Stretch"/>
                </Style>
            </ListBox.ItemContainerStyle>
        </ListBox>
    </Border>
</Window>

The code-behind that supplies a list should look like this:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Windows;
using System.Windows.Controls;

namespace VSPTest
{
    public partial class Window1 : Window
    {
        private class DataItem
        {
            public DataItem(int index)
            {
                this.index = index;
            }

            private readonly int index;

            public int Index {
                get {
                    return index;
                }
            }

            public override string ToString()
            {
                return index.ToString();
            }
        }

        private class MyTestCollection : IList<DataItem>
        {
            public MyTestCollection(int count)
            {
                this.count = count;
            }

            private readonly int count;

            public DataItem this[int index] {
                get {
                    var result = new DataItem(index);
                    System.Diagnostics.Debug.WriteLine("ADD " + result.ToString());
                    return result;
                }
                set {
                    throw new NotImplementedException();
                }
            }

            public int Count {
                get {
                    return count;
                }
            }

            public bool IsReadOnly {
                get {
                    throw new NotImplementedException();
                }
            }

            public int IndexOf(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void Insert(int index, Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void RemoveAt(int index)
            {
                throw new NotImplementedException();
            }

            public void Add(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void Clear()
            {
                throw new NotImplementedException();
            }

            public bool Contains(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void CopyTo(Window1.DataItem[] array, int arrayIndex)
            {
                throw new NotImplementedException();
            }

            public bool Remove(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public IEnumerator<Window1.DataItem> GetEnumerator()
            {
                for (int i = 0; i < count; i++) {
                    yield return this[i];
                }
            }

            System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
            {
                return this.GetEnumerator();
            }
        }

        public Window1()
        {
            InitializeComponent();

            DataContext = new MyTestCollection(10000);
        }

        void ListBox_CleanUpVirtualizedItem(object sender, CleanUpVirtualizedItemEventArgs e)
        {
            System.Diagnostics.Debug.WriteLine("DEL " + e.Value.ToString());
        }
    }
}

So, this displays an application with a ListBox, which is forced to virtualize its items with the IsVirtualizing attached property. It takes its items from the data context, for which a custom IList<T> implementation is supplied that creates 10000 data items on the fly (when they are retrieved via the indexer).

For debugging purposes, the text ADD # (where # equals the item index) is output whenever an item is created, and the CleanUpVirtualizedItem event is used to output DEL # when an item goes out of view and its UI is released by the virtualizing stack panel.

Now, my wish is that my custom list implementation supplies items upon request - in this minimal sample, by creating them on the fly, and in the real project by loading them from the database. Unfortunately, VirtualizingStackPanel does not seem to behave this way - instead, it invokes the enumerator of the list upon program start and first retrieves all 10000 items!

Thus, my question is: How can I use VirtualizingStackPanel for actual virtualization of data (as in, not loading all the data) rather than just reducing the number of GUI elements?

Is there any way to tell the virtualizing stack panel how many items there are in total and telling it to access them by index as needed, rather than using the enumerator? (Like, for example, the Delphi Virtual TreeView component works, if I recall correctly.)
Are there any ingenious ways of capturing the event when an item actually comes into view, so at least I could normally just store a unique key of each item and only load the remaining item data when it is requested? (That would seem like a hacky solution, though, as I would still have to provide the full-length list for no real reason, other than satisfying the WPF API.)
Is another WPF class more suitable for this kind of virtualization?

EDIT: Following dev hedgehog's advice, I have created a custom ICollectionView implementation. Some of its methods are still implemented to throw NotImplementedExceptions, but the ones that get called when the window is opened do not.

However, it seems that about the first thing that is called for that collection view is the GetEnumerator method, enumerating all 10000 elements again (as evidenced by the debug output, where I print a message for every 1000th item), which is what I was trying to avoid.

Here is an example to reproduce the issue:

Window1.xaml

<Window x:Class="CollectionViewTest.Window1"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="CollectionViewTest" Height="300" Width="300"
    >
    <Border Padding="5">
        <ListBox VirtualizingStackPanel.IsVirtualizing="True" ItemsSource="{Binding .}">
            <ListBox.ItemTemplate>
                <DataTemplate>
                    <Border BorderBrush="Blue" BorderThickness="2" CornerRadius="5" Margin="2" Padding="4" Background="Chocolate">
                        <Border BorderBrush="Red" BorderThickness="1" CornerRadius="4" Padding="3" Background="Yellow">
                            <TextBlock Text="{Binding Index}"/>
                        </Border>
                    </Border>
                </DataTemplate>
            </ListBox.ItemTemplate>
            <ListBox.ItemContainerStyle>
                <Style TargetType="{x:Type ListBoxItem}">
                    <Setter Property="HorizontalContentAlignment" Value="Stretch"/>
                </Style>
            </ListBox.ItemContainerStyle>
        </ListBox>
    </Border>
</Window>

Window1.xaml.cs

using System;
using System.ComponentModel;
using System.Collections;
using System.Collections.Specialized;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Windows;

namespace CollectionViewTest
{
    public partial class Window1 : Window
    {
        private class DataItem
        {
            public DataItem(int index)
            {
                this.index = index;
            }

            private readonly int index;

            public int Index {
                get {
                    return index;
                }
            }

            public override string ToString()
            {
                return index.ToString();
            }
        }

        private class MyTestCollection : IList<DataItem>
        {
            public MyTestCollection(int count)
            {
                this.count = count;
            }

            private readonly int count;

            public DataItem this[int index] {
                get {
                    var result = new DataItem(index);
                    if (index % 1000 == 0) {
                        System.Diagnostics.Debug.WriteLine("ADD " + result.ToString());
                    }
                    return result;
                }
                set {
                    throw new NotImplementedException();
                }
            }

            public int Count {
                get {
                    return count;
                }
            }

            public bool IsReadOnly {
                get {
                    throw new NotImplementedException();
                }
            }

            public int IndexOf(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void Insert(int index, Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void RemoveAt(int index)
            {
                throw new NotImplementedException();
            }

            public void Add(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void Clear()
            {
                throw new NotImplementedException();
            }

            public bool Contains(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public void CopyTo(Window1.DataItem[] array, int arrayIndex)
            {
                throw new NotImplementedException();
            }

            public bool Remove(Window1.DataItem item)
            {
                throw new NotImplementedException();
            }

            public IEnumerator<Window1.DataItem> GetEnumerator()
            {
                for (int i = 0; i < count; i++) {
                    yield return this[i];
                }
            }

            System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
            {
                return this.GetEnumerator();
            }
        }

        private class MyCollectionView : ICollectionView
        {
            public MyCollectionView(int count)
            {
                this.list = new MyTestCollection(count);
            }

            private readonly MyTestCollection list;

            public event CurrentChangingEventHandler CurrentChanging;

            public event EventHandler CurrentChanged;

            public event NotifyCollectionChangedEventHandler CollectionChanged;

            public System.Globalization.CultureInfo Culture {
                get {
                    return System.Globalization.CultureInfo.InvariantCulture;
                }
                set {
                    throw new NotImplementedException();
                }
            }

            public IEnumerable SourceCollection {
                get {
                    return list;
                }
            }

            public Predicate<object> Filter {
                get {
                    throw new NotImplementedException();
                }
                set {
                    throw new NotImplementedException();
                }
            }

            public bool CanFilter {
                get {
                    return false;
                }
            }

            public SortDescriptionCollection SortDescriptions {
                get {
                    return new SortDescriptionCollection();
                }
            }

            public bool CanSort {
                get {
                    throw new NotImplementedException();
                }
            }

            public bool CanGroup {
                get {
                    throw new NotImplementedException();
                }
            }

            public ObservableCollection<GroupDescription> GroupDescriptions {
                get {
                    return new ObservableCollection<GroupDescription>();
                }
            }

            public ReadOnlyObservableCollection<object> Groups {
                get {
                    throw new NotImplementedException();
                }
            }

            public bool IsEmpty {
                get {
                    throw new NotImplementedException();
                }
            }

            public object CurrentItem {
                get {
                    return null;
                }
            }

            public int CurrentPosition {
                get {
                    throw new NotImplementedException();
                }
            }

            public bool IsCurrentAfterLast {
                get {
                    throw new NotImplementedException();
                }
            }

            public bool IsCurrentBeforeFirst {
                get {
                    throw new NotImplementedException();
                }
            }

            public bool Contains(object item)
            {
                throw new NotImplementedException();
            }

            public void Refresh()
            {
                throw new NotImplementedException();
            }

            private class DeferRefreshObject : IDisposable
            {
                public void Dispose()
                {
                }
            }

            public IDisposable DeferRefresh()
            {
                return new DeferRefreshObject();
            }

            public bool MoveCurrentToFirst()
            {
                throw new NotImplementedException();
            }

            public bool MoveCurrentToLast()
            {
                throw new NotImplementedException();
            }

            public bool MoveCurrentToNext()
            {
                throw new NotImplementedException();
            }

            public bool MoveCurrentToPrevious()
            {
                throw new NotImplementedException();
            }

            public bool MoveCurrentTo(object item)
            {
                throw new NotImplementedException();
            }

            public bool MoveCurrentToPosition(int position)
            {
                throw new NotImplementedException();
            }

            public IEnumerator GetEnumerator()
            {
                return list.GetEnumerator();
            }
        }

        public Window1()
        {
            InitializeComponent();
            this.DataContext = new MyCollectionView(10000);
        }
    }
}

VirtualizingStackPanel has nothing to do with Data Virtualization. You need to implement that yourself on the data level — Fede
– Fede, Commented Feb 11, 2014 at 19:58
@HighCore: I see - so I will have to write my own ItemsControl-like UI control? — O. R. Mapper
– O. R. Mapper, Commented Feb 11, 2014 at 20:30
When you say fairly large amount. How large? Do you want to discard items after they are out of view. Round trips to the database are expensive. Running a query multiple times is expensive. Are you running out of memory? I don't get moving a load to a database and network to save memory. — paparazzo
– paparazzo, Commented Feb 11, 2014 at 20:30
@Blam: Maybe 10000 items, maybe a million, maybe more. Ultimately, it depends on the users' filtering parameters. What can be said for sure, though, is that especially with so many items, only a few dozens will probably ever be in view. Hence, there is absolutely no point in loading all the other items when all the UI basically needs to know is their total number (for scrolling). In particular, I don't see a reason to risk blocking the UI while unnecessarily loading thousands of items, when those items will probably never be displayed (but are just "there" in the perception of the user ... — O. R. Mapper
– O. R. Mapper, Commented Feb 11, 2014 at 20:35
..., as he or she can reach them by simply scrolling down in the list, if he or she tries). — O. R. Mapper
– O. R. Mapper, Commented Feb 11, 2014 at 20:35

123 456 789 0 · Accepted Answer · 2014-02-11 20:06:16Z

4

You want Data Virtualization, you have UI Virtualization right now.

You can take a look more about data virtualization here

answered Feb 11, 2014 at 20:06

123 456 789 0

10.8k4 gold badges46 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

demonllama · Accepted Answer · 2018-06-25 16:08:52Z

To get around the issue where the VirtualizingStackPanel attempts to enumerate over its entire data source, I stepped through the source code on http://referencesource.microsoft.com (https://referencesource.microsoft.com/#PresentationFramework/src/Framework/System/Windows/Controls/VirtualizingStackPanel.cs)

I'll provide the TLDR here:

If you specified VirtualizingStackPanel.ScrollUnit="Pixel" you need to make sure all the items displayed/virtualized from its ItemTemplate are the same size (height). Even if you are a pixel different, all bets are off and you'll most likely trigger a loading of the whole list.
If the items being displayed are not exactly the same height, you must specify VirtualizingStackPanel.ScrollUnit="Item".

My Findings:

There are several 'landmines' in the VirtualizingStackPanel source that trigger an attempt to iterate over the entire collection via the index operator []. One of these is during the Measurement cycle of which it attempts to update the virtualized container size to make the scrollviewer accurate. If any new items being added during this cycle aren't the same size when in Pixel mode, it iterates over the whole list to adjust and you are hosed.

Another 'landmine' has something to do with selection and triggering a hard refresh. This is applicable more for grids - but under the hood, its using a DataGridRowPresenter which derives from VirtualizingStackPanel. Because it wants to keep selections in sync between refreshing, it attempts to enumerate all. This means we need to disable selection (keep in mind that clicking a row triggers a selection).

I solved this by deriving my own grid and overriding OnSelectionChanged:

protected override void OnSelectionChanged(SelectionChangedEventArgs e)
{
    if(SelectedItems.Count > 0)
    {
        UnselectAll();
    }
    e.Handled = true;
}

There seems to be other gotchas, but I haven't been able to reliably trigger them yet. The real 'fix' would be to roll our own VirtualizingStackPanel with looser constraints for generating the containersize. After all, for large datasets (million+), the accuracy of scrollbar matters much less. If I have time to do this, I'll update my answer with the gist/github repo.

In my tests I used a data virtualization solution available here: https://github.com/anagram4wander/VirtualizingObservableCollection.

dev hedgehog · Accepted Answer · 2014-02-11 20:04:51Z

1

You are almost there just it is not the VirtualizingStackPanel that invokes the enumerator of the list.

When you Binding to ListBox.ItemsSource there will be a ICollectionView interface created automatically between your actual Source of data and ListBox Target. That interface is the meany invoking the enumerator.

How to fix this? Well just write your own CollectionView class that inherits from ICollectionView interface. Pass it to ItemsSource and ListBox will know you wish to have your own view of data. Which is what you sort of need. Then once ListBox realizes you are using your own view, just return the needed data when requested by ListBox. That would be it. Play nice with ICollectionView :)

answered Feb 11, 2014 at 20:04

dev hedgehog

8,8414 gold badges33 silver badges55 bronze badges

4 Comments

O. R. Mapper Over a year ago

That sounds like a promising approach, however, after implementing a minimal version of ICollectionView (i.e. adding method implementations until no NotImplementedExceptions are thrown any more), the GetEnumerator method of the ICollectionView implementation gets called, still enumerating all 10000 items. Also, looking at ICollectionView, I cannot see any members that would allow to retrieve any specific items, or their total number.

O. R. Mapper Over a year ago

So, it seems like even with ICollectionView, any items displays just have to enumerate the full list.

dev hedgehog Over a year ago

I cant really say without code. However I did it with ICollectionView interface when I had to write data virtualization for TreeView. It should work. Check it again or post us the code you are using. Upload the project somewhere online

O. R. Mapper Over a year ago

I have edited my question with an example that uses a collection view. GetEnumerator is called and all 10000 items are retrieved right upon launching the application, rather than gradually, as they come into view.

Marek Fekete · Accepted Answer · 2017-08-08 13:17:27Z

Long time after the question was posted, but may be useful to someone out there. While solving the exact same problem, I found out that your ItemsProvider (in your case, MyTestCollection,) has to implement IList interface (non-templated). Only then the VirtualizingStackPanel accesses the individual items via [] operator, rather than enumerating them via GetEnumerator. In your case, it should be enough to add:

    object IList.this[int index]
    {
        get { return this[index]; }
        set { throw new NotSupportedException(); }
    }

    public int IndexOf(DataItem item)
    {
        // TODO: Find a good way to find out the item's index
        return DataItem.Index;
    }

    public int IndexOf(object value)
    {
        var item = value as DataItem;
        if (item != null)
            return IndexOf(item);
        else
            throw new NullReferenceException();
    }

All the remaining IList's members can be left unimplemented, as far as I can see.

Collectives™ on Stack Overflow

VirtualizingStackPanel with Virtualized List

4 Answers 4

Comments

Comments

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related