Performance with large data sources

Apr 7, 2010 at 10:33 PM

I've been using D3 linegraphs successfully with moderately large data sources (20K points) but now have an application that needs to display around 1M data points (3hours of 100Hz data).

Initially plotting the chart & redrawing in response to changes in the plot size takes around 5 seconds on an i7 based PC with a GeForce GT 330M graphics card.

There isn't much point displaying 1M data points when all data is displayed but when the user zooms in to a smaller region of the plot the high sample frequency data becomes useful.

I've considered showing the overview at 1Hz (approx 10K points) and then changing to higher frequency data (e.g. 100Hz) as the user zooms in.

Has anyone else faced a similar problem and what solution (if any) did you come up with?

Apr 11, 2010 at 9:14 AM
Edited Oct 24, 2013 at 8:58 PM
Hi there,  I just found D3 by googling for high performance charting libraries and it looks good - yes I have a requirement similar to the above - 1m data points. When I was a student I wrote a charting control in Windows Forms that could handle around 1m datapoints for a real-time data acquisition system using resampling. Hope it helps and good luck to you!  
Apr 11, 2010 at 7:00 PM
Edited Oct 24, 2013 at 8:59 PM
... On inspection D3 might already be doing this. I just had a look at LineGraph/FrequencyFilter classes in the D3 source. In the UpdateCore method, the points in the dataseries are first filtered using a class FrequencyFilter (resampled?) and then cropped using class FakePointList to render only what is in the viewable area, prior to being rendered in OnRender. The FrequencyFilter appears to be doing some sort of sampling to reduce the dataset when large points sets are being rendered (correct me if Im wrong!).
Apr 12, 2010 at 4:22 AM

Andy,

Thanks for your input.  I'll work through your suggestions and post my findings.

Grant.

Apr 12, 2010 at 9:21 AM
Edited Oct 24, 2013 at 9:00 PM
Great, I'd be interested to see them!
Apr 13, 2010 at 3:17 AM

not sure if this addresses the above problem, but below is a filter I use to reduce the number of points to display, by specifying the number of data points to skip.  hope it helps someone.

 

    public class GraphPointFilter : PointsFilterBase {
        private int skipNumber = 0;

        public GraphPointFilter(int _skipNumber) {
            skipNumber = _skipNumber;
        }

        /// <summary>
        /// The number of points to skip from the original set
        /// </summary>
        public int SkipNumber {
            get { return skipNumber; }
            set { skipNumber = value; }
        }

        public override List<Point> Filter(List<Point> points) {
            if (skipNumber == 0) {
                return points;
            }

            List<Point> resultPoints = new List<Point>();

            int i = 0;

            foreach (var element in points) {
                if (i % (skipNumber+1) == 0) {
                    resultPoints.Add(element);
                }

                i++;
            }

            return resultPoints;
        }

        public void Notify() {
            RaiseChanged();
        }
    }

 

 

e.