You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Jun 13, 2026 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275382421230,3sleep342-21ksoftirqd/319:15:113
280162221160,5sleep00-21swapper/021:30:000
280319421100,3sleep20-21swapper/221:35:012
283807421080,4sleep23056-21PK-Backend23:15:012
275066127546,20sleep30-21swapper/319:05:233
275106127437,13sleep10-21swapper/119:07:271
275021127050,14sleep00-21swapper/019:05:170
274981426744,15sleep20-21swapper/219:05:112
275139899403,24cyclictest2863963-21taskset00:25:392
275139899383,20cyclictest0-21swapper/200:16:352
275139899374,30cyclictest2755684-21cpuspeed_turbos19:20:112
275139899373,20cyclictest0-21swapper/219:38:272
275139899372,20cyclictest0-21swapper/220:33:212
275139899372,20cyclictest0-21swapper/219:34:432
275139899372,20cyclictest0-21swapper/219:34:432
275139899371,21cyclictest0-21swapper/220:13:162
275139899362,19cyclictest0-21swapper/222:18:272
275139899362,19cyclictest0-21swapper/221:08:022
275139899362,19cyclictest0-21swapper/220:40:352
275139899361,20cyclictest0-21swapper/219:51:112
275139899361,20cyclictest0-21swapper/219:10:532
275139899353,19cyclictest0-21swapper/220:08:112
275139899352,19cyclictest0-21swapper/223:30:492
275139899352,19cyclictest0-21swapper/220:15:572
275139899343,19cyclictest0-21swapper/220:23:192
275139899342,20cyclictest0-21swapper/222:50:352
275139899332,19cyclictest0-21swapper/221:18:432
275139899332,19cyclictest0-21swapper/219:49:592
275139899290,17cyclictest2821991-21chrt22:27:422
275139899290,17cyclictest2821991-21chrt22:27:422
275139899285,21cyclictest1-21systemd23:20:002
275139899281,22cyclictest0-21swapper/221:00:112
28346022270,10sleep013-21ksoftirqd/023:05:010
275139899272,21cyclictest2861119-21apt00:20:112
275139899272,21cyclictest0-21swapper/220:50:092
275139899272,19cyclictest0-21swapper/223:45:102
2751398992711,14cyclictest1-21systemd00:00:012
275139899268,16cyclictest2769406-21dump-pmu-power20:00:002
275139899267,17cyclictest2868094-21ls00:40:012
275139899261,21cyclictest2799630-21apt-get21:25:032
2751398992610,14cyclictest2806630-21grep21:45:012
139099260,4rtkit-daemon1389-21rtkit-daemon22:56:291
275139899256,17cyclictest918-21rs:main2
275139899255,18cyclictest2782737-21cat20:35:202
275139899254,19cyclictest2778791-21cat20:25:132
275139899252,21cyclictest2856591-21strings00:05:222
275139899252,19cyclictest2824187-21apt22:35:092
275139899252,19cyclictest0-21swapper/223:35:122
275139899251,22cyclictest2762916-21cpuspeed_turbos19:40:102
275139899251,21cyclictest2819166-21cpuspeed_turbos22:20:122
2751396992515,8cyclictest2807600-21ntp_states21:45:181
27817452240,4sleep13056-21PK-Backend20:35:011
275139899247,15cyclictest2770178-21cut20:00:162
275139899246,16cyclictest2847607-21seq23:40:132
275139899246,16cyclictest2822752-21expr22:30:102
275139899242,20cyclictest2828965-21chrt22:48:152
275139899242,19cyclictest2816383-21python322:10:192
275139899242,18cyclictest0-21swapper/222:40:112
275139899241,21cyclictest2789603-21python320:55:212
275139899241,20cyclictest2852171-21munin-run23:55:002
275139899241,19cyclictest2857653-21apt00:10:112
2751398992410,12cyclictest2809307-21grep21:50:192
275139899240,3cyclictest436-20systemd-journal00:30:322
275139899240,3cyclictest2758783-21unin-run19:30:002
2751396992421,2cyclictest3056-21PK-Backend21:03:061
28508992230,3sleep03056-21PK-Backend23:50:120
2751401992319,3cyclictest42-21ksoftirqd/300:25:003
2751401992316,5cyclictest2859100-21cron00:14:593
2751401992314,7cyclictest2864349-21unin-run00:30:003
2751401992312,10cyclictest2752331-21ntp_states19:10:173
275140199230,21cyclictest2796789-21python321:15:203
275139899235,16cyclictest2854249-21ls00:00:152
275139899233,18cyclictest918-21rs:main2
275139899231,19cyclictest2794142-21cnf-update-db21:10:022
275139899231,19cyclictest2785249-21/usr/sbin/munin20:45:132
275139899231,18cyclictest2753819-21apt19:15:102
2751398992310,11cyclictest2834979-21cpuspeed_turbos23:05:122
275139699230,21cyclictest3056-21PK-Backend00:01:111
2751391992320,2cyclictest0-21swapper/022:27:400
2751391992320,2cyclictest0-21swapper/022:27:400
28111002220,3sleep20-21swapper/221:55:222
2751401992220,1cyclictest0-21swapper/321:14:183
2751401992219,2cyclictest3056-21PK-Backend21:43:533
275140199221,8cyclictest2819852-21python322:20:213
2751401992217,4cyclictest42-21ksoftirqd/323:35:013
2751401992217,4cyclictest42-21ksoftirqd/321:10:013
275140199220,21cyclictest3056-21PK-Backend21:37:573
275139899225,15cyclictest2803588-21cstates21:35:122
275139899221,18cyclictest2833975-21python323:00:192
2751396992220,1cyclictest3056-21PK-Backend00:15:241
2751396992219,2cyclictest3056-21PK-Backend21:22:401
2751396992217,3cyclictest2804857-21dump-pmu-power21:40:001
275139699220,21cyclictest0-21swapper/123:13:321
275139699220,20cyclictest3056-21PK-Backend22:54:331
275139699220,20cyclictest3056-21PK-Backend21:28:201
275139699220,20cyclictest3056-21PK-Backend19:48:581
275139699220,20cyclictest3056-21PK-Backend00:36:541
2751391992220,1cyclictest0-21swapper/023:09:320
2751391992220,1cyclictest0-21swapper/022:54:370
2751391992220,1cyclictest0-21swapper/022:12:470
2751391992220,1cyclictest0-21swapper/022:01:580
2751391992220,1cyclictest0-21swapper/021:41:100
2751391992220,1cyclictest0-21swapper/021:04:130
2751391992220,1cyclictest0-21swapper/020:05:050
2751391992220,1cyclictest0-21swapper/019:39:360
2751391992220,1cyclictest0-21swapper/019:32:270
2751391992220,1cyclictest0-21swapper/019:32:260
2751391992220,1cyclictest0-21swapper/000:35:430
2751391992217,3cyclictest2820447-21sh22:25:000
275139199220,21cyclictest0-21swapper/020:53:110
275140199219,10cyclictest2769443-21unin-run20:00:013
2751401992119,1cyclictest3056-21PK-Backend23:10:323
2751401992119,1cyclictest3056-21PK-Backend22:16:043
2751401992119,1cyclictest3056-21PK-Backend21:53:403
2751401992119,1cyclictest0-21swapper/322:26:523
2751401992119,1cyclictest0-21swapper/322:26:513
2751401992119,1cyclictest0-21swapper/321:01:143
2751401992119,1cyclictest0-21swapper/320:58:103
2751401992119,1cyclictest0-21swapper/320:23:223
2751401992119,1cyclictest0-21swapper/320:17:143
2751401992119,1cyclictest0-21swapper/320:12:073
2751401992118,2cyclictest3056-21PK-Backend23:00:453
2751401992118,2cyclictest3056-21PK-Backend22:55:513
2751401992116,4cyclictest42-21ksoftirqd/300:00:013
2751401992114,6cyclictest42-21ksoftirqd/323:55:013
275140199210,19cyclictest3056-21PK-Backend23:24:113
275140199210,19cyclictest3056-21PK-Backend23:24:103
275140199210,19cyclictest3056-21PK-Backend21:57:323
275140199210,19cyclictest3056-21PK-Backend21:30:143
275139899218,11cyclictest2840587-21latency23:20:162
275139899218,11cyclictest2840587-21latency23:20:162
275139899215,14cyclictest0-21swapper/222:00:212
2751398992119,1cyclictest0-21swapper/222:55:402
2751398992119,1cyclictest0-21swapper/221:49:422
2751396992119,1cyclictest3056-21PK-Backend23:55:291
2751396992119,1cyclictest3056-21PK-Backend21:13:551
2751396992119,1cyclictest3056-21PK-Backend20:51:311
2751396992119,1cyclictest3056-21PK-Backend20:23:321
2751396992119,1cyclictest3056-21PK-Backend20:17:381
2751396992119,1cyclictest3056-21PK-Backend00:06:531
2751396992119,1cyclictest0-21swapper/123:25:361
2751396992119,1cyclictest0-21swapper/123:16:361
2751396992119,1cyclictest0-21swapper/122:13:511
2751396992119,1cyclictest0-21swapper/121:42:141
2751396992119,1cyclictest0-21swapper/121:34:161
2751396992119,1cyclictest0-21swapper/120:40:431
2751396992118,2cyclictest3056-21PK-Backend20:27:381
275139699210,20cyclictest3056-21PK-Backend23:51:381
275139699210,19cyclictest3056-21PK-Backend22:40:061
275139699210,19cyclictest3056-21PK-Backend22:35:131
275139699210,19cyclictest3056-21PK-Backend20:11:431
2751391992119,1cyclictest0-21swapper/022:37:410
2751391992119,1cyclictest0-21swapper/022:18:420
2751391992119,1cyclictest0-21swapper/021:48:060
2751391992119,1cyclictest0-21swapper/021:39:060
2751391992119,1cyclictest0-21swapper/020:56:150
2751391992119,1cyclictest0-21swapper/019:48:010
2751391992119,1cyclictest0-21swapper/019:25:300
2751391992119,1cyclictest0-21swapper/019:23:280
2751391992119,1cyclictest0-21swapper/019:18:340
2751391992119,1cyclictest0-21swapper/000:25:090
2751391992119,1cyclictest0-21swapper/000:08:390
275139199210,20cyclictest0-21swapper/023:11:350
275139199210,1cyclictest0-21swapper/021:59:560
275139199210,1cyclictest0-21swapper/021:54:010
275139199210,19cyclictest3056-21PK-Backend20:41:070
275140199206,5cyclictest42-21ksoftirqd/300:16:363
2751401992018,1cyclictest3056-21PK-Backend23:19:173
2751401992018,1cyclictest3056-21PK-Backend23:05:393
2751401992018,1cyclictest3056-21PK-Backend21:48:463
2751401992018,1cyclictest0-21swapper/323:47:503
2751401992018,1cyclictest0-21swapper/323:41:543
2751401992018,1cyclictest0-21swapper/323:37:023
2751401992018,1cyclictest0-21swapper/322:45:513
2751401992018,1cyclictest0-21swapper/321:20:273
2751401992018,1cyclictest0-21swapper/320:50:123
2751401992018,1cyclictest0-21swapper/320:48:093
2751401992018,1cyclictest0-21swapper/320:28:293
2751401992018,1cyclictest0-21swapper/319:34:083
2751401992018,1cyclictest0-21swapper/319:34:083
2751401992018,1cyclictest0-21swapper/300:00:413
2751401992015,4cyclictest42-21ksoftirqd/319:40:003
2751401992012,6cyclictest2857509-21sed00:10:003
275140199201,17cyclictest2764479-21apt19:45:113
275140199200,1cyclictest3056-21PK-Backend23:25:123
275140199200,19cyclictest0-21swapper/322:52:483
275140199200,19cyclictest0-21swapper/320:43:033
275140199200,19cyclictest0-21swapper/320:00:243
275140199200,19cyclictest0-21swapper/300:30:353
2751398992018,1cyclictest0-21swapper/222:09:302
275139699203,15cyclictest2751652-21sort19:10:111
2751396992018,1cyclictest3056-21PK-Backend23:47:461
2751396992018,1cyclictest3056-21PK-Backend21:18:481
2751396992018,1cyclictest3056-21PK-Backend20:47:391
2751396992018,1cyclictest3056-21PK-Backend19:19:241
2751396992018,1cyclictest0-21swapper/123:23:331
2751396992018,1cyclictest0-21swapper/123:23:331
2751396992018,1cyclictest0-21swapper/122:07:561
2751396992017,2cyclictest3056-21PK-Backend23:41:041
2751396992017,2cyclictest3056-21PK-Backend20:04:331
2751396992017,2cyclictest3056-21PK-Backend19:20:261
2751396992017,2cyclictest3056-21PK-Backend00:12:341
2751396992013,5cyclictest2835939-21sshd23:05:281
275139699200,1cyclictest3056-21PK-Backend00:20:041
275139699200,19cyclictest3056-21PK-Backend22:18:441
275139699200,19cyclictest0-21swapper/122:02:011
275139699200,18cyclictest3056-21PK-Backend22:46:491
275139699200,18cyclictest3056-21PK-Backend22:22:351
2751391992018,1cyclictest0-21swapper/022:47:420
2751391992018,1cyclictest0-21swapper/022:30:450
2751391992018,1cyclictest0-21swapper/020:36:010
2751391992018,1cyclictest0-21swapper/020:10:260
2751391992018,1cyclictest0-21swapper/000:31:510
275139199200,1cyclictest3056-21PK-Backend21:05:130
275139199200,1cyclictest0-21swapper/023:28:170
275139199200,1cyclictest0-21swapper/020:00:460
275139199200,1cyclictest0-21swapper/019:55:250
275139199200,19cyclictest0-21swapper/023:47:030
275139199200,19cyclictest0-21swapper/022:06:520
275139199200,18cyclictest3056-21PK-Backend23:30:200
275140199194,4cyclictest42-21ksoftirqd/322:35:013
2751401991917,1cyclictest3056-21PK-Backend22:11:103
2751401991917,1cyclictest0-21swapper/319:52:593
2751401991915,3cyclictest42-21ksoftirqd/300:39:203
275140199190,1cyclictest3056-21PK-Backend22:09:083
275140199190,1cyclictest0-21swapper/322:39:433
275140199190,1cyclictest0-21swapper/319:26:103
2751396991917,1cyclictest3056-21PK-Backend23:33:191
2751396991917,1cyclictest3056-21PK-Backend22:27:291
2751396991917,1cyclictest3056-21PK-Backend22:27:291
2751396991916,2cyclictest3056-21PK-Backend20:59:151
2751396991916,1cyclictest3056-21PK-Backend20:38:391
2751396991911,6cyclictest2808573-21cat21:50:101
275139699190,1cyclictest3056-21PK-Backend21:08:001
275139699190,1cyclictest3056-21PK-Backend20:07:361
275139699190,1cyclictest3056-21PK-Backend19:26:071
275139699190,1cyclictest3056-21PK-Backend00:27:331
275139699190,1cyclictest0-21swapper/121:57:081
2751391991917,1cyclictest0-21swapper/023:56:020
2751391991917,1cyclictest0-21swapper/023:42:090
2751391991917,1cyclictest0-21swapper/023:20:340
2751391991917,1cyclictest0-21swapper/023:20:330
2751391991917,1cyclictest0-21swapper/000:21:170
2751391991917,1cyclictest0-21swapper/000:13:330
2751391991917,1cyclictest0-21swapper/000:03:460
2751391991911,6cyclictest2780850-21awk20:30:190
2751391991911,6cyclictest2399-21gdbus23:18:030
275139199190,1cyclictest0-21swapper/000:19:140
275140199182,12cyclictest2781812-21apt20:35:113
2751401991816,1cyclictest0-21swapper/319:42:333
275140199181,3cyclictest141rcu_preempt22:02:183
2751401991812,4cyclictest2771387-21/usr/sbin/munin20:05:143
275140199181,15cyclictest2800404-21ntp_states21:25:163
275140199181,14cyclictest978-21snmpd19:20:213
275140199180,16cyclictest2825920-21apt22:40:103
2751396991816,1cyclictest3056-21PK-Backend23:37:121
2751396991816,1cyclictest3056-21PK-Backend19:35:541
2751396991813,3cyclictest2866053-21latency_hist00:35:001
2751391991816,1cyclictest0-21swapper/021:28:190
275140199179,7cyclictest42-21ksoftirqd/320:30:133
2751396991710,5cyclictest2833973-21seq23:00:201
2751391991715,1cyclictest3056-21PK-Backend21:10:210
2751391991715,1cyclictest0-21swapper/022:40:450
2751391991714,2cyclictest3056-21PK-Backend20:15:320
2751391991711,5cyclictest2786199-21sed20:45:200
2751391991711,4cyclictest2752166-21load19:10:150
2751391991710,6cyclictest2797436-21systemd21:20:000
2751391991710,5cyclictest2845696-21grep23:35:110
2751391991710,5cyclictest2799127-21apt-get21:25:000
2751391991710,5cyclictest2776607-21awk20:20:090
2751391991611,3cyclictest2767184-21timerandwakeup19:50:200
275139199161,11cyclictest2831344-21apt-get22:55:040
275139699158,6cyclictest2764205-21cron19:45:001
275139199158,5cyclictest2778641-21expr20:25:110
2751396991412,1cyclictest3056-21PK-Backend19:32:021
2751396991412,1cyclictest3056-21PK-Backend19:32:021
2751396991410,3cyclictest3056-21PK-Backend22:31:341
275139199149,3cyclictest1095-21runrttasks19:40:020
2751396991210,1cyclictest3056-21PK-Backend19:51:151
275139699119,1cyclictest3056-21PK-Backend19:55:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional