You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 08:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Nov 19, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
339139099124109,7cyclictest0-21swapper/022:35:020
3391390998325,17cyclictest3405907-21cat23:45:030
3391390997430,40cyclictest0-21swapper/020:36:350
3391390997139,4cyclictest3397347-21modprobe21:03:340
3391390997139,4cyclictest3397347-21modprobe21:03:340
3391390996965,2cyclictest0-21swapper/019:30:020
3391398996810,14cyclictest3398417-21modprobe21:25:002
339139899672,19cyclictest235-21systemd-journal21:55:002
339139899665,16cyclictest3397615-21modprobe21:09:592
3391398996628,33cyclictest3397512-21latency_hist21:05:022
3391398996628,33cyclictest3397512-21latency_hist21:05:022
3391390996628,9cyclictest1-21systemd00:01:350
3391390996544,16cyclictest3400764-21munin-run22:10:020
3391390996532,28cyclictest3395804-21cut20:35:010
3391390996532,28cyclictest3395804-21cut20:35:000
3391390996528,32cyclictest3237915-1kworker/0:2H19:35:030
3391390996528,32cyclictest3237915-1kworker/0:2H19:35:020
3391390996523,36cyclictest3399427-21modprobe21:40:020
339139899646,9cyclictest3399175-21modprobe21:40:002
3391398996415,44cyclictest0-21swapper/223:20:012
3391398996414,10cyclictest235-21systemd-journal21:00:012
3391390996421,39cyclictest3402018-21sshd22:31:360
3391390996421,38cyclictest3399188-21latency_hist21:40:020
3391398996334,23cyclictest30-21ksoftirqd/222:00:002
339139899633,16cyclictest3395356-21wc20:25:012
3391398996321,6cyclictest306-21in:imuxsock20:01:342
3391398996321,6cyclictest306-21in:imuxsock20:01:332
3391390996344,15cyclictest3402813-21sshd22:48:340
3391398996213,12cyclictest235-21systemd-journal21:20:012
3391390996248,10cyclictest3391806-21sshd19:16:330
3391398996121,6cyclictest3395724-21latency_hist20:30:032
339139899612,13cyclictest3396855-21latency_hist20:55:012
3391398996118,11cyclictest3399752-21cat21:50:012
3391395996119,10cyclictest23-21ksoftirqd/123:55:021
3391390996153,4cyclictest3400229-21munin-run22:00:000
3391390996145,13cyclictest3404891-21sshd23:27:370
3391390996145,11cyclictest3393144-21sshd19:43:320
3391390996142,15cyclictest3393689-21sshd19:51:560
3391390996142,14cyclictest3399938-21sshd21:51:350
3391390996127,30cyclictest3393942-21sshd19:56:330
339139899603,17cyclictest3392934-21latency_hist19:40:012
3391398996026,5cyclictest3391881-21sh19:20:012
3391398996025,4cyclictest3403173-21latency_hist22:55:022
3391398996025,4cyclictest3403173-21latency_hist22:55:022
3391398996024,5cyclictest3406049-21latency_hist23:50:012
3391398996022,12cyclictest1-21systemd19:50:012
3391398996022,12cyclictest1-21systemd19:50:012
3391398996015,12cyclictest3403777-21cut23:05:022
3391390996046,11cyclictest3397585-21sshd21:05:340
3391390996044,11cyclictest3391583-21munin-run19:15:000
3391390996040,15cyclictest363-21snmpd21:33:420
3391390996031,9cyclictest1-21systemd22:20:010
339139899595,22cyclictest3405407-21cron23:40:002
3391398995923,8cyclictest3395836-21latency_hist20:35:012
3391398995923,8cyclictest3395836-21latency_hist20:35:012
3391398995918,8cyclictest3402554-21sshd22:42:262
3391390995947,9cyclictest3398662-21sshd21:29:360
3391390995946,10cyclictest3404338-21sshd23:15:350
3391390995946,10cyclictest3393386-21sshd19:46:330
3391390995946,10cyclictest3393386-21sshd19:46:330
3391390995945,11cyclictest3406419-21sshd23:55:340
3391390995944,12cyclictest3397865-21sshd21:13:360
3391390995944,11cyclictest3396297-21sshd20:40:340
3391390995944,11cyclictest3395275-21sshd20:23:330
3391390995942,13cyclictest3394479-21sshd20:07:320
3391390995941,14cyclictest3405461-21munin-run23:40:010
339139899585,15cyclictest3403326-21cron23:00:002
339139899585,13cyclictest3396062-21grep20:40:002
339139899584,18cyclictest3407786-21munin-run00:25:012
339139899583,9cyclictest0-21swapper/200:40:002
3391398995826,28cyclictest3406990-21latency_hist00:10:022
3391398995825,4cyclictest3394569-21latency_hist20:10:012
3391398995822,4cyclictest3404708-21latency_hist23:25:012
3391398995814,7cyclictest0-21swapper/223:30:012
3391390995843,12cyclictest3394730-21sshd20:12:340
3391390995743,10cyclictest3403616-21munin-run23:05:010
3391390995742,11cyclictest3403379-21/usr/sbin/munin23:00:010
3391390995742,10cyclictest3408600-21munin-run00:40:010
3391390995737,16cyclictest3399689-21sshd21:47:140
3391390995737,15cyclictest363-21snmpd00:26:280
3391390995737,15cyclictest363-21snmpd00:26:270
3391390995722,30cyclictest3396734-21latency_hist20:50:020
3391390995722,30cyclictest3396734-21latency_hist20:50:020
339139899566,17cyclictest3401274-21munin-run22:20:002
339139899563,19cyclictest3400742-21modprobe22:10:002
3391390995648,5cyclictest3397068-21sshd20:58:350
3391390995644,9cyclictest3396814-21sshd20:53:350
3391390995643,9cyclictest3398407-21sshd21:23:350
3391390995642,10cyclictest3404636-21sshd23:22:340
3391390995642,10cyclictest3401017-21munin-run22:15:010
3391390995642,10cyclictest3400512-21munin-run22:05:000
3391398995524,8cyclictest30-21ksoftirqd/223:32:352
3391398995524,5cyclictest3393992-21sendmail20:00:002
3391395995537,13cyclictest235-21systemd-journal20:54:591
3391390995546,5cyclictest3392847-21sshd19:35:330
3391390995543,8cyclictest3406201-21munin-run23:55:010
3391390995542,9cyclictest363-21snmpd20:16:160
3391390995542,9cyclictest3402564-21sshd22:44:360
3391390995542,10cyclictest3392128-21munin-run19:25:000
3391390995531,20cyclictest235-21systemd-journal22:55:020
3391390995531,20cyclictest235-21systemd-journal22:55:010
3391390995514,10cyclictest3395596-21cat20:30:010
3391398995435,14cyclictest272-21dbus-daemon22:50:012
339139899542,17cyclictest3392441-21latency_hist19:30:012
3391390995444,5cyclictest3407515-21munin-run00:20:010
3391390995443,7cyclictest3407247-21munin-run00:15:000
3391390995442,9cyclictest3405131-21sshd23:30:350
3391390995441,9cyclictest363-21snmpd00:21:270
3391390995436,13cyclictest3394226-21sshd20:02:340
3391390995436,13cyclictest3394226-21sshd20:02:330
339140299538,11cyclictest37-21ksoftirqd/323:40:033
3391402995311,6cyclictest131rcu_preempt19:50:013
3391402995311,6cyclictest131rcu_preempt19:50:013
339139899533,15cyclictest3408427-21cat00:35:012
3391398995328,15cyclictest235-21systemd-journal21:15:002
3391390995346,4cyclictest3404085-21sshd23:10:350
3391390995346,4cyclictest3398128-21sshd21:18:350
3391390995343,6cyclictest3401528-21sshd22:24:340
3391390995330,17cyclictest3391510-21grep19:10:030
3391398995221,9cyclictest30-21ksoftirqd/223:45:012
3391390995239,10cyclictest3401797-21munin-run22:30:000
3391390995233,15cyclictest3405919-21sshd23:46:340
339140299514,9cyclictest37-21ksoftirqd/323:30:023
3391390995144,4cyclictest3403834-21sshd23:05:360
3391390995144,4cyclictest3403834-21sshd23:05:350
3391402995032,14cyclictest272-21dbus-daemon20:20:013
3391402995032,13cyclictest235-21systemd-journal00:05:003
3391398995019,8cyclictest30-21ksoftirqd/219:25:012
3391390995043,4cyclictest3408269-21sshd00:33:360
339140299498,12cyclictest37-21ksoftirqd/323:45:013
3391402994910,6cyclictest131rcu_preempt19:35:023
3391402994910,6cyclictest131rcu_preempt19:35:013
3391390994936,9cyclictest363-21snmpd00:06:120
3391402994834,10cyclictest3406474-21tr23:59:593
3391402994832,9cyclictest235-21systemd-journal20:05:013
3391402994832,9cyclictest235-21systemd-journal20:05:003
3391402994831,11cyclictest37-21ksoftirqd/322:25:003
3391402994815,5cyclictest131rcu_preempt20:39:593
3391398994837,6cyclictest3395003-21cron20:20:002
339139899475,9cyclictest1-21systemd20:50:022
339139899475,9cyclictest1-21systemd20:50:012
3391398994731,10cyclictest306-21in:imuxsock00:03:352
3391398994717,9cyclictest131rcu_preempt00:00:032
3391395994714,6cyclictest23-21ksoftirqd/119:36:431
339140299463,9cyclictest37-21ksoftirqd/300:15:003
3391402994629,13cyclictest37-21ksoftirqd/320:55:013
3391398994637,6cyclictest3407248-21grep00:15:012
3391398994621,17cyclictest272-21dbus-daemon22:05:002
3391395994631,12cyclictest235-21systemd-journal20:32:051
3391395994631,12cyclictest235-21systemd-journal20:32:041
3391395994631,11cyclictest235-21systemd-journal23:32:371
3391395994628,14cyclictest3400193-21sshd21:57:151
339140299457,6cyclictest131rcu_preempt20:10:013
3391402994529,12cyclictest272-21dbus-daemon20:00:003
3391402994528,13cyclictest235-21systemd-journal23:25:003
3391402994517,25cyclictest37-21ksoftirqd/322:35:023
3391402994512,6cyclictest131rcu_preempt22:15:013
3391398994532,8cyclictest306-21in:imuxsock21:45:012
3391395994538,5cyclictest221rcuc/119:34:211
3391395994538,5cyclictest221rcuc/119:34:211
3391395994526,14cyclictest272-21dbus-daemon00:15:011
339140299448,5cyclictest131rcu_preempt20:50:003
339140299448,5cyclictest131rcu_preempt20:50:003
3391402994429,11cyclictest3403221-21wc22:55:033
3391402994429,11cyclictest3403221-21wc22:55:023
339139899445,25cyclictest3398934-21cut21:35:002
3391395994433,8cyclictest3398396-21sshd21:21:361
339140299437,8cyclictest37-21ksoftirqd/322:50:023
339140299435,10cyclictest37-21ksoftirqd/320:45:013
3391402994330,9cyclictest37-21ksoftirqd/319:25:003
3391402994327,12cyclictest272-21dbus-daemon21:35:003
339140299431,13cyclictest37-21ksoftirqd/323:10:013
339140299431,13cyclictest37-21ksoftirqd/323:10:013
3391402994310,6cyclictest131rcu_preempt23:50:003
3391395994328,11cyclictest23-21ksoftirqd/121:10:001
3391395994310,5cyclictest0-21swapper/100:40:001
3391402994226,12cyclictest272-21dbus-daemon20:25:003
3391398994224,12cyclictest235-21systemd-journal20:45:002
339139599428,6cyclictest0-21swapper/100:00:411
339139599425,6cyclictest23-21ksoftirqd/122:15:021
3391395994230,9cyclictest3391867-21sendmail19:20:001
3391395994228,10cyclictest3393755-21latency_hist19:55:011
3391395994225,13cyclictest3393994-21tr20:00:011
339140299416,8cyclictest37-21ksoftirqd/321:05:013
339140299416,8cyclictest37-21ksoftirqd/321:05:013
3391402994123,13cyclictest235-21systemd-journal21:40:013
3391402994111,6cyclictest131rcu_preempt20:30:013
339139599419,8cyclictest0-21swapper/121:35:081
3391395994129,9cyclictest23-21ksoftirqd/121:00:011
3391395994125,11cyclictest272-21dbus-daemon22:20:001
3391395994124,13cyclictest3402835-21munin-run22:50:001
3391395994123,14cyclictest23-21ksoftirqd/100:20:001
3391395994120,15cyclictest272-21dbus-daemon23:59:591
339140299405,8cyclictest37-21ksoftirqd/319:30:003
339140299404,9cyclictest37-21ksoftirqd/300:35:013
339140299404,6cyclictest131rcu_preempt23:15:013
339140299404,6cyclictest131rcu_preempt19:55:003
3391402994031,7cyclictest37-21ksoftirqd/322:10:013
3391402994030,8cyclictest37-21ksoftirqd/300:30:023
3391402994030,8cyclictest37-21ksoftirqd/300:30:013
3391402994029,9cyclictest37-21ksoftirqd/322:29:593
3391402994027,9cyclictest306-21in:imuxsock20:57:353
3391402994027,8cyclictest235-21systemd-journal23:20:003
3391402994011,5cyclictest131rcu_preempt23:55:023
339139899404,25cyclictest3392634-21munin-plugin-st19:35:012
339139899404,25cyclictest3392634-21munin-plugin-st19:35:002
3391395994027,8cyclictest272-21dbus-daemon20:20:011
3391395994026,10cyclictest3396036-21sshd20:39:351
3391395994023,12cyclictest363-21snmpd20:06:071
3391395994023,12cyclictest23-21ksoftirqd/122:25:351
3391395994020,15cyclictest3392381-21munin-run19:30:011
339139599401,34cyclictest23-21ksoftirqd/123:20:021
339139599401,28cyclictest3397596-21kworker/u8:021:55:011
339140299395,7cyclictest37-21ksoftirqd/319:20:003
3391402993930,6cyclictest361rcuc/323:35:003
3391402993929,8cyclictest37-21ksoftirqd/322:20:023
3391402993926,9cyclictest306-21in:imuxsock21:17:003
3391402993910,10cyclictest37-21ksoftirqd/300:25:003
3391398993926,10cyclictest3403849-21sshd23:08:362
3391398993926,10cyclictest3403849-21sshd23:08:352
3391398993921,13cyclictest1-21systemd19:55:012
3391398993920,13cyclictest1-21systemd22:25:022
339139599399,6cyclictest0-21swapper/123:35:581
339139599398,6cyclictest0-21swapper/120:03:531
339139599398,6cyclictest0-21swapper/120:03:531
339139599394,7cyclictest23-21ksoftirqd/123:12:351
3391395993926,8cyclictest23-21ksoftirqd/121:15:001
3391395993925,11cyclictest3400834-21sed22:10:011
3391395993925,10cyclictest3395261-21sshd20:20:331
3391395993924,12cyclictest272-21dbus-daemon22:55:021
3391395993924,12cyclictest272-21dbus-daemon22:55:011
3391395993924,10cyclictest235-21systemd-journal21:20:011
3391395993923,13cyclictest3405915-21sshd23:45:351
3391395993922,14cyclictest363-21snmpd21:41:251
3391395993922,13cyclictest1-21systemd00:35:011
3391395993910,4cyclictest0-21swapper/122:36:041
339140299389,5cyclictest131rcu_preempt22:40:023
339140299388,7cyclictest37-21ksoftirqd/300:08:093
339140299383,10cyclictest37-21ksoftirqd/300:20:003
3391402993828,7cyclictest37-21ksoftirqd/321:30:023
339140299382,5cyclictest131rcu_preempt23:05:013
339140299382,5cyclictest131rcu_preempt23:00:003
3391398993822,13cyclictest30-21ksoftirqd/219:45:002
3391398993810,6cyclictest131rcu_preempt22:12:342
339139599385,6cyclictest23-21ksoftirqd/121:35:001
3391395993831,4cyclictest0-21swapper/119:10:011
3391395993826,8cyclictest23-21ksoftirqd/119:50:021
3391395993826,8cyclictest23-21ksoftirqd/119:50:021
3391395993825,10cyclictest3391596-21munin-run19:15:011
3391395993824,10cyclictest3398679-21munin-run21:30:001
3391395993823,11cyclictest3406994-21munin-run00:10:001
3391395993820,13cyclictest3408038-21munin-run00:30:011
3391395993820,13cyclictest3408038-21munin-run00:30:001
3391395993819,13cyclictest272-21dbus-daemon22:35:001
3391395993816,18cyclictest3392141-21munin-run19:25:001
3391395993810,8cyclictest0-21swapper/122:40:211
3391402993730,5cyclictest37-21ksoftirqd/300:37:053
3391402993728,4cyclictest3394753-21munin-plugin-st20:15:013
3391402993725,9cyclictest37-21ksoftirqd/322:05:003
3391402993724,9cyclictest235-21systemd-journal21:59:353
3391402993722,11cyclictest235-21systemd-journal20:31:343
3391402993722,11cyclictest235-21systemd-journal20:31:333
3391402993721,11cyclictest3392896-21systemctl19:39:593
339139599378,7cyclictest0-21swapper/121:48:201
339139599376,7cyclictest0-21swapper/123:04:301
339139599376,6cyclictest0-21swapper/120:29:441
3391395993724,10cyclictest3396447-21sed20:45:031
3391395993723,11cyclictest3403317-21sshd22:58:341
3391395993723,10cyclictest3397367-21munin-run21:05:011
3391395993723,10cyclictest3397367-21munin-run21:05:001
3391395993722,10cyclictest3407758-21sshd00:22:351
3391395993721,11cyclictest3403882-21munin-run23:10:021
3391395993721,11cyclictest3403882-21munin-run23:10:011
3391402993628,6cyclictest37-21ksoftirqd/321:55:033
3391402993627,6cyclictest361rcuc/319:15:013
3391402993626,7cyclictest37-21ksoftirqd/321:42:353
3391402993624,10cyclictest37-21ksoftirqd/321:22:113
339139899363,27cyclictest3402312-21cut22:40:012
3391398993626,7cyclictest3406199-21munin-plugin-st23:55:002
3391395993623,9cyclictest23-21ksoftirqd/122:25:011
3391395993623,9cyclictest23-21ksoftirqd/119:45:021
3391395993621,11cyclictest23-21ksoftirqd/120:15:001
3391395993621,10cyclictest48-21kcompactd020:48:211
3391395993621,10cyclictest48-21kcompactd020:48:201
3391402993522,9cyclictest361rcuc/321:10:003
339139899355,9cyclictest30-21ksoftirqd/200:16:572
3391398993521,9cyclictest1-21systemd00:30:012
3391398993521,9cyclictest1-21systemd00:30:002
3391395993525,7cyclictest23-21ksoftirqd/123:25:031
3391395993520,11cyclictest23-21ksoftirqd/123:44:591
3391395993519,11cyclictest272-21dbus-daemon23:30:001
3391402993423,8cyclictest3397851-21sshd21:10:363
3391402993422,9cyclictest37-21ksoftirqd/319:45:023
3391398993417,10cyclictest3402049-21modprobe22:35:002
3391395993421,9cyclictest48-21kcompactd022:00:281
3391398993320,9cyclictest306-21in:imuxsock20:15:002
3391398993317,9cyclictest306-21in:imuxsock19:15:002
3391402993220,10cyclictest37-21ksoftirqd/322:45:013
3391398993225,5cyclictest412-21sshd21:26:342
3391398993217,11cyclictest235-21systemd-journal22:27:212
3391402993120,9cyclictest37-21ksoftirqd/321:50:013
3391398993122,7cyclictest30-21ksoftirqd/223:14:592
3391402992921,6cyclictest37-21ksoftirqd/319:10:023
339139899293,16cyclictest3391399-21latency_hist19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional