Fill a List with Large Amount of Data

Fill a List with Large Amount of Data 

2013.02.06 by Ovidiu Cucu

In my opinion, making database queries in separate threads in order to avoid UI blocking is the last option a Windows developer should have. Of course, that’s an option like any others but, to avoid headaches of thread synchronization and so on, we have to consider first the following:

• 1. Optimize the database, index fields if necessary.
• 2. Choose the right database access technology.

Well, let’s say we are just “poor developers” and both database designer and project manager are idiots. Then, there’s no choice and we have to live with them without resolving #1 and #2. In most cases, UI is blocked as a result of populating a list control with a large number of rows, from a recordset. Have we to make this in a separate thread? Not really, wait a minute!

• 3. Use a virtual list. A virtual list does not keep the UI busy until it’s filled with thousands, tens of thousands or even hundreds of thousands items. It just asks info about items which have to be currently displayed.
• 4. Not enough? We have to deal with queries which return millions, tens of millions or even more results? No problem, can use a cache mechanism.

Both #3 and #4 are supported in the Windows common listview (SysListView32) control.

Inserting a large number of items in a listview control, may take a lot of time.
A virtual list does not internally keep items information (like text and so on). Instead, it notifies its parent, asking info only for the items/subitems which are currently displayed. That makes it a good choyce if dealing with a large number of items.

Using a virtual list

1. Set LVS_OWNERDATA style.
   First note this style cannot be set after control creation. For a listview control contained in a dialog resource, it can be done by setting True for Owner Data property. For a listview control kept by a CListView class, the right place is in the overridden PreCreateWindow. 

    

   BOOL CVirtualListView::PreCreateWindow(CREATESTRUCT& cs)
   {
       cs.style &= ~LVS_TYPEMASK; // clear old type
       cs.style |= LVS_REPORT;    // make it report type
       cs.style |= LVS_OWNERDATA; // make it virtual list
   
      return CListView::PreCreateWindow(cs);
   }

2. Load data.
   May be, for example, an array of database records as a result of an SQL query. See the attached demo project for a concrete example.
3. Set the virtual list items count.

   UINT CVirtualListView::FillList(Recordset& rs)
   {
       // ...
       // Load data in m_arrRows array instead of actually fill the list control
       // ...
       CListCtrl& listCtrl = GetListCtrl();     // get listview control
       const int nCount = m_arrRows.GetCount(); // get number of rows
       listCtrl.SetItemCountEx(nCount);         // set list items count
       return nCount;
   }

4. Finally, handle the LVN_GETDISPINFO notification.
   As said in the beginning, a virtual list doesn’t internally keep items info. Instead, it sends LVN_GETDISPINFO notification via WM_NOTIFY message, each time it needs info about the items to be actually displayed.
   For a listview control we can use the wizard to map LVN_GETDISPINFO in the parent dialog class or the reflected =LVN_GETDISPINFO notification, in a class derived from CListCtrl.
   In a class derived from CListView we can also use the wizard to map reflected =LVN_GETDISPINFO notification. 

    

   // VirtualListView.h
   // ...
   class CVirtualListView : public CListView
   {
      // ...
       DECLARE_MESSAGE_MAP()
       afx_msg void OnLvnGetdispinfo(NMHDR *pNMHDR, LRESULT *pResult);
   };

   First argument is a pointer to a NMLVDISPINFO structure. From its member of type LVITEM, we can get which info is required (text, image, etc) and the index of item and subitem that is being displayed. Next, we have to fill that structure with the information from an external data collection.

   // VirtualListView.cpp
       // ...
       ON_NOTIFY_REFLECT(LVN_GETDISPINFO, &CVirtualListView::OnLvnGetdispinfo)
   END_MESSAGE_MAP()
   // ...
   void CVirtualListView::OnLvnGetdispinfo(NMHDR *pNMHDR, LRESULT *pResult)
   {
       NMLVDISPINFO *pDispInfo = reinterpret_cast<NMLVDISPINFO*>(pNMHDR);
       LV_ITEM* pItem = &(pDispInfo)->item;
   
      if(pItem->mask & LVIF_TEXT)
      {
         // Set the text for the given item 
         // Use pItem->iItem, pItem->iSubItem, pItem->pszText, and pItem->cchTextMax.
         CDBRecord* pRow = m_arrRows.GetAt(pItem->iItem); // recordset row -> item index
         CDBValue* pValue = pRow->GetAt(pItem->iSubItem); // recordset column -> subitem index
         _tcscpy_s(pItem->pszText, pItem->cchTextMax, pValue->GetString());
      }
      if(pItem->mask & LVIF_IMAGE)
      {
         // ...set an index in an imagelist 
      }
      // etc.
      *pResult = 0;
   }

Demo project

Download: Virtual_List_Demo.zip (2192 downloads)

The demo project uses the same SQL query to fill a non-virtual and a virtual listview. As can be seen in the screenshot, the virtual list worked over ten times faster.

Using Virtual List – Demo Application

 

Additional notes

• The demo application uses an SQL Server Compact 4.0 database. Its runtime engine usually comes with Visual Studio 2012. Anyway, if it isn’t already installed on your machine, you can download it for free, from Microsoft Download Center:
  http://www.microsoft.com/en-us/download/details.aspx?id=17876
• A future article will show how to improve the virtual listview control in order to deal with much larger amount of data, by caching.

Resources

• MSDN: CListCtrl Class
• MSDN: CListView Class
• MSDN: Virtual List Controls
• MSDN: LVN_GETDISPINFO notification code
• MSDN: NMLVDISPINFO structure

See also