[logback-dev] benchmarking extension to junit?

[Jörn Huxhorn]

On Mon, Aug 4, 2008 at 10:21 PM, Ceki Gulcu <listid at qos.ch> wrote:

> Joern,
>
> Thanks for your input. Instead of forcing the test to sleep, another
> approach is


The sleep was just meant as a placeholder for the code that will take some
time, i.e. the actual code of the test. So I constructed one case where the
test will work for sure, and one test that would always fail because it
takes longer than the expected amount of time.
I probably should have added some comments :)


> to adjust the expected performance according to the capabilities of the
> host
> CPU. This is the approach adopted in BogoPerf. See
>
>
> http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/logback/classic/LoggerPerfTest.java
>
> http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/logback/classic/util/BogoPerf.java
>

I'll take a look for sure.

Joern.



> Joern Huxhorn wrote:
> > Ceki Gulcu wrote:
> >> Hello all,
> >>
> >> Logback contains a small number of time-sensitive tests. The time needed
> to
> >> execute these tests depend on the capabilities of the host CPU. For
> example, if
> >> on my machine a test executes in 1 millisecond, I would expect it to
> >> finish in say 1*10 milliseconds on most machine -- allowing for 10 fold
> variation.
> >>
> >> Does anyone know of a junit extension which can help me normalize the
> >> coefficient, 10 in the previous example, so that it adapts to the
> >> speed of the host CPU? There is also the problem of the JIT
> >> compiler...
> >>
> >> Any recommendations on the subject?
> >>
> >>
> > Well, I don't have a real solution to your problem...
> > The only performance-related JUnit framework I'm aware of is
> > http://clarkware.com/software/JUnitPerf.html but I think it's outdated
> > and I never used it.
> > It could be useful for load-testing, though.
> >
> > If you are using JUnit 4.x I'd suggest something like the following:
> >
> > import org.junit.Test;
> > import org.junit.BeforeClass;
> > import org.junit.Before;
> > import static junit.framework.Assert.assertEquals;
> > import static junit.framework.Assert.fail;
> >
> > public class TimingTest
> > {
> >
> >     private static long referenceTime;
> >
> >     @BeforeClass
> >     public static void initReferenceTime()
> >     {
> >         long time=System.currentTimeMillis();
> >
> >         // lets waste some time...
> >         StringBuffer msg=new StringBuffer();
> >         for(int i=0;i<10000;i++)
> >         {
> >             msg.append(i);
> >         }
> >         System.out.println(msg.toString());
> >         referenceTime=System.currentTimeMillis()-time;
> >         System.out.println("ReferenceTime: "+referenceTime+" millis");
> >     }
> >
> >     @Test(timeout = 1000)
> >     public void works()
> >         throws Exception
> >     {
> >         long time=System.currentTimeMillis();
> >         Thread.sleep(referenceTime*5);
> >         time=System.currentTimeMillis()-time;
> >         long maxTime=referenceTime*10;
> >         if(time>maxTime)
> >         {
> >             fail("Test was not expected to take more than "+maxTime+"
> > millis but took "+time+" millis!");
> >         }
> >     }
> >
> >     @Test(timeout = 1000)
> >     public void fails()
> >         throws Exception
> >     {
> >         long time=System.currentTimeMillis();
> >         Thread.sleep(referenceTime*15);
> >         time=System.currentTimeMillis()-time;
> >         long maxTime=referenceTime*10;
> >         if(time>maxTime)
> >         {
> >             fail("Test was not expected to take more than "+maxTime+"
> > millis but took "+time+" millis!");
> >         }
> >     }
> > }
> >
> >
> > The problem with this approach, however, is that referenceTime isn't
> > very stable. It fluctuates between 15ms and 40ms on my development
> > system. I guess this fluctuation could be reduced by using an even
> > higher number in the initReferenceTime loop.
> > The time-wasting would have to be changed depending on the
> > problem-domain of the real test, e.g. something like the above if String
> > operations are performed in the actual code that is tested.
> >
> > The timeout I've given in @Test is supposed to be a definitive upper
> > level that should never be reached. This could be left out, though.
> >
> > I have some doubts what the Hotspot-JIT will be doing if the above
> > initReferenceTime is used the same way in more than one class. I have
> > some hope that it does NOT realize that it's the same code but I'm not
> sure.
> >
> > The code above isn't perfect, neither is it especially elegant, but it's
> > probably good enough for your use case if initReferenceTime and the
> > factors are tuned a bit...
> > I don't know.
> >
> > Joern.
> > _______________________________________________
> > logback-dev mailing list
> > logback-dev at qos.ch
> > http://qos.ch/mailman/listinfo/logback-dev
>
> --
> Ceki Gülcü
>
> QOS.ch is looking to hire talented developers located in Switzerland
> to work on cutting-edge software projects. If you think you are
> qualified, then please contact ceki at qos.ch.
> _______________________________________________
> logback-dev mailing list
> logback-dev at qos.ch
> http://qos.ch/mailman/listinfo/logback-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://qos.ch/pipermail/logback-dev/attachments/20080804/738392af/attachment.htm