You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 09:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jun 05, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24073210087,9sleep00-21swapper/019:07:400
2301429582,8sleep20-21swapper/219:05:182
2394529165,9sleep30-21swapper/319:06:043
2418628669,13sleep10-21swapper/119:09:101
24348992726,1cyclictest0-21swapper/322:30:403
2434899271,1cyclictest141rcu_preempt00:25:363
2434099270,3cyclictest0-21swapper/021:38:160
2434899250,1cyclictest0-21swapper/323:33:413
2434899250,0cyclictest0-21swapper/322:35:103
2434099250,5cyclictest14547-21grep19:55:170
2434099250,5cyclictest14547-21grep19:55:160
2434599240,22cyclictest0-21swapper/221:23:542
24342992422,1cyclictest5262-21gltestperf00:35:191
2434299242,19cyclictest30195-21apt-get22:35:121
2434899230,0cyclictest0-21swapper/321:47:333
2434299230,23cyclictest0-21swapper/121:31:501
2434099235,16cyclictest1-21systemd21:55:270
2434099232,2cyclictest2559-21cron20:40:000
2434099231,1cyclictest141rcu_preempt21:15:350
2434099231,0cyclictest0-21swapper/021:40:390
2434099230,4cyclictest0-21swapper/021:26:020
2434899221,18cyclictest0-21swapper/323:10:133
2434899220,22cyclictest0-21swapper/321:37:383
2434899220,1cyclictest0-21swapper/323:38:333
2434599220,0cyclictest0-21swapper/223:19:342
2434099220,5cyclictest0-21swapper/022:02:280
2434099220,4cyclictest0-21swapper/022:54:120
2434099220,3cyclictest0-21swapper/000:02:200
2434899210,21cyclictest0-21swapper/323:03:123
24342992121,0cyclictest0-21swapper/123:01:091
2434099211,2cyclictest0-21swapper/022:48:460
2434099210,7cyclictest0-21swapper/023:13:490
2434899202,1cyclictest141rcu_preempt22:19:463
2434599201,1cyclictest141rcu_preempt23:11:062
2434599201,1cyclictest141rcu_preempt22:48:302
2434599200,18cyclictest0-21swapper/223:02:562
2434299200,20cyclictest0-21swapper/122:01:061
2434299200,0cyclictest0-21swapper/122:32:201
2434099200,2cyclictest0-21swapper/023:17:160
2434099200,2cyclictest0-21swapper/022:17:250
2434099200,1cyclictest0-21swapper/021:30:390
2434899191,1cyclictest141rcu_preempt00:04:043
2434899190,1cyclictest0-21swapper/321:25:003
2434299190,2cyclictest0-21swapper/122:50:051
2434299190,2cyclictest0-21swapper/100:20:141
2434299190,19cyclictest0-21swapper/100:10:231
2434099193,4cyclictest0-21swapper/021:54:120
2434099193,4cyclictest0-21swapper/021:54:110
2434099190,1cyclictest0-21swapper/021:45:190
2434899183,6cyclictest0-21swapper/300:20:153
24348991817,1cyclictest0-21swapper/323:48:033
24348991817,1cyclictest0-21swapper/323:05:463
24348991813,2cyclictest0-21swapper/321:22:053
2434899180,1cyclictest0-21swapper/300:38:133
2434599180,1cyclictest141rcu_preempt00:21:422
24342991818,0cyclictest0-21swapper/122:26:161
24342991817,1cyclictest0-21swapper/123:08:581
24342991813,5cyclictest2283-21sh23:35:191
2434299180,4cyclictest0-21swapper/123:20:211
2434099188,6cyclictest27580-21taskset20:21:260
2434099182,4cyclictest3124-21cpuspeed_turbos20:40:140
2434099181,2cyclictest17347-21diskmemload23:04:570
2434099180,6cyclictest0-21swapper/000:31:040
2434099180,2cyclictest0-21swapper/019:15:110
24348991713,1cyclictest0-21swapper/323:29:303
2434899170,3cyclictest0-21swapper/321:33:513
2434899170,0cyclictest0-21swapper/322:59:463
2434599174,1cyclictest25112-21apt-get23:30:122
2434599170,14cyclictest0-21swapper/200:38:132
24342991716,1cyclictest0-21swapper/100:18:591
2434299170,1cyclictest0-21swapper/100:27:151
2434299170,17cyclictest0-21swapper/100:04:421
2434099175,6cyclictest1405-21idleruntime-cro19:29:590
2434099171,1cyclictest4431-21apt-get00:35:140
24340991710,6cyclictest17137-21grep23:05:220
2434099170,7cyclictest1538-21perf23:55:010
2434099170,3cyclictest0-21swapper/022:35:570
2434099170,3cyclictest0-21swapper/000:10:450
2434099170,2cyclictest0-21swapper/000:22:120
24348991614,1cyclictest39-21ksoftirqd/322:12:543
2434899160,16cyclictest0-21swapper/321:56:433
2434599161,4cyclictest2061-21diskmemload23:55:132
24345991614,1cyclictest537-21snmpd23:41:552
2434599161,2cyclictest141rcu_preempt21:54:372
2434599161,2cyclictest141rcu_preempt21:54:362
2434599161,14cyclictest0-21swapper/222:34:332
2434599161,13cyclictest0-21swapper/221:48:082
2434599160,1cyclictest0-21swapper/221:58:292
2434599160,15cyclictest0-21swapper/223:20:212
2434299160,1cyclictest0-21swapper/123:44:451
2434299160,1cyclictest0-21swapper/122:59:461
2434099169,6cyclictest25285-21head19:10:160
2434099169,6cyclictest16232-21ssh23:25:000
2434099168,7cyclictest17343-21cut20:00:210
2434099164,11cyclictest10012-21perf21:24:590
24340991615,0cyclictest0-21swapper/021:00:130
2434099161,4cyclictest0-21swapper/020:15:140
2434099161,3cyclictest0-21swapper/023:35:160
2434099161,3cyclictest0-21swapper/022:12:320
2434099161,1cyclictest0-21swapper/022:40:130
2434099160,5cyclictest0-21swapper/023:49:260
2434899158,4cyclictest0-21swapper/322:51:043
2434899151,6cyclictest1-21systemd00:07:233
2434899151,4cyclictest17347-21diskmemload22:26:263
24348991514,1cyclictest0-21swapper/321:51:193
24348991514,1cyclictest0-21swapper/321:51:183
2434899150,1cyclictest0-21swapper/323:57:543
2434599156,7cyclictest32-21ksoftirqd/222:05:152
24345991513,1cyclictest0-21swapper/221:13:192
2434599151,13cyclictest26734-21df21:15:152
24345991511,3cyclictest17800-21cat20:00:272
2434299152,12cyclictest0-21swapper/121:56:311
24342991514,1cyclictest0-21swapper/121:14:211
24342991510,4cyclictest3415-21apache222:18:481
2434299150,10cyclictest0-21swapper/100:30:001
2434299150,0cyclictest0-21swapper/123:48:021
2434099158,6cyclictest3413-21cut00:15:150
2434099157,7cyclictest23578-21grep00:07:260
2434099157,7cyclictest13171-21sed19:50:250
2434099157,2cyclictest29916-21chrt20:25:580
2434099156,3cyclictest500-21sleep23:35:010
2434099151,13cyclictest23110-21awk20:15:010
24340991510,4cyclictest18983-21wc21:10:190
2434099150,2cyclictest0-21swapper/022:28:120
2434099150,1cyclictest0-21swapper/022:25:000
2434899141,8cyclictest20201-21runrttasks22:48:183
24348991413,1cyclictest0-21swapper/300:11:333
2434899140,7cyclictest0-21swapper/319:45:123
2434599148,1cyclictest141rcu_preempt23:50:132
2434299149,4cyclictest18327-21sh21:30:001
2434299141,3cyclictest17154-21/usr/sbin/munin20:00:201
24342991413,1cyclictest0-21swapper/122:07:411
24342991412,1cyclictest0-21swapper/121:45:171
2434299140,4cyclictest0-21swapper/121:36:571
2434299140,1cyclictest141rcu_preempt23:12:431
2434299140,1cyclictest0-21swapper/123:15:191
2434099148,5cyclictest21580-21ssh00:25:240
2434099148,5cyclictest16502-21ssh23:25:020
2434099148,5cyclictest15269-21grep21:05:140
2434099147,6cyclictest31755-21expr22:55:160
2434099142,2cyclictest225-21systemd-journal20:55:000
24340991410,2cyclictest3327-21tail19:30:250
24340991410,2cyclictest10491-21sed19:45:250
2434099140,4cyclictest225-21systemd-journal20:35:000
2434099140,3cyclictest0-21swapper/022:30:430
2434899137,3cyclictest18509-21diskstats21:10:163
2434899136,4cyclictest31659-21sh21:18:093
2434899131,4cyclictest0-21swapper/323:51:503
24348991312,1cyclictest11960-21ssh23:21:403
24348991310,2cyclictest0-21swapper/321:42:113
2434899130,7cyclictest0-21swapper/300:18:293
2434899130,3cyclictest0-21swapper/323:40:013
2434599139,3cyclictest5031-21apt20:44:592
2434599139,3cyclictest23082-21cron20:15:012
2434599139,3cyclictest17383-21cat23:05:262
2434599133,9cyclictest10368-21/usr/sbin/munin00:00:172
24345991310,2cyclictest7281-21wc22:40:192
24345991310,2cyclictest4649-21turbostat22:20:012
24345991310,2cyclictest24036-21sh22:50:202
24345991310,2cyclictest12352-21cron20:59:592
24345991310,2cyclictest11370-21grep21:25:182
2434599130,2cyclictest0-21swapper/222:25:042
2434599130,1cyclictest0-21swapper/200:10:152
2434299139,3cyclictest25-21ksoftirqd/120:45:131
2434299138,4cyclictest21090-21ssh22:48:591
2434299132,10cyclictest0-21swapper/122:10:371
2434099139,2cyclictest11369-21tr20:55:170
2434099136,2cyclictest2349-21tr23:55:160
24340991310,2cyclictest6638-21cut19:40:140
24340991310,2cyclictest30790-21wc19:20:260
24340991310,2cyclictest14463-21grep22:05:270
2434099130,4cyclictest16565-21sshd23:44:270
2434899129,2cyclictest23078-21seq20:15:003
2434899129,2cyclictest16885-21tail21:05:253
2434899127,3cyclictest0-21swapper/322:20:413
2434899123,3cyclictest7053-21hwlatdetect22:40:173
2434899121,8cyclictest0-21swapper/319:55:133
2434899121,8cyclictest0-21swapper/319:55:123
2434899121,1cyclictest0-21swapper/323:17:453
2434899120,9cyclictest0-21swapper/320:30:113
2434899120,1cyclictest0-21swapper/322:01:003
2434899120,11cyclictest29640-21cat20:25:253
2434899120,11cyclictest12550-21tail19:50:193
2434599129,2cyclictest8594-21head20:50:162
2434599129,2cyclictest5424-21find19:35:212
2434599129,2cyclictest31791-21tr19:25:132
2434599129,2cyclictest30734-21tr20:30:132
2434599129,2cyclictest29362-21grep20:25:212
2434599129,2cyclictest22795-21sed22:10:262
2434599129,2cyclictest19147-21grep00:05:162
2434599129,2cyclictest18890-21grep20:05:142
2434599129,2cyclictest13477-21grep21:00:162
2434599129,2cyclictest1301-21tr23:35:142
2434599125,3cyclictest225-21systemd-journal19:17:282
2434599124,4cyclictest225-21systemd-journal22:25:002
24345991212,0cyclictest0-21swapper/222:57:012
2434599120,2cyclictest17347-21diskmemload21:30:312
2434599120,1cyclictest0-21swapper/222:39:542
2434599120,1cyclictest0-21swapper/222:04:532
2434299129,2cyclictest8578-21ssh22:21:331
2434299129,2cyclictest7819-21rm21:42:321
2434299129,2cyclictest6719-21sort19:40:141
2434299129,2cyclictest26975-21ssh23:50:181
2434299129,2cyclictest21383-21ls20:10:131
2434299129,2cyclictest13624-21sed21:00:171
2434299129,2cyclictest11101-21cat19:50:011
2434299128,3cyclictest11708-21grep19:50:141
2434299124,3cyclictest0-21swapper/100:06:311
2434299122,9cyclictest3738-21kworker/1:222:40:131
2434299121,5cyclictest0-21swapper/119:55:131
2434299121,5cyclictest0-21swapper/119:55:121
24342991211,1cyclictest0-21swapper/120:50:191
24342991210,1cyclictest24814-21expr19:10:141
2434299120,11cyclictest26909-21grep21:15:181
2434299120,11cyclictest0-21swapper/121:22:041
2434099129,2cyclictest6005-21tail20:45:150
2434099125,1cyclictest203952sleep020:06:020
2434899119,2cyclictest5010-21cron20:44:593
2434899119,2cyclictest3204-21sed19:30:253
2434899119,2cyclictest32028-21cat19:25:143
2434899119,1cyclictest8289-21cat20:50:133
2434899119,1cyclictest5422-21wget20:45:143
2434899119,1cyclictest415-21apache_accesses20:35:123
2434899119,1cyclictest28898-21rm00:30:123
2434899118,2cyclictest6956-21cat19:40:153
2434899118,2cyclictest29307-21chrt19:20:063
2434899118,2cyclictest25338-21grep19:10:163
2434899114,5cyclictest12499-21apt-get22:05:143
2434899112,6cyclictest23357-21apt-get20:15:113
2434899110,2cyclictest141rcu_preempt20:55:163
2434899110,11cyclictest27505-21chrt20:20:283
2434899110,10cyclictest4119-21cut19:35:123
2434599119,1cyclictest22103-21awk00:25:292
2434599118,2cyclictest7638-21cut20:50:002
2434599118,2cyclictest4522-21grep21:40:212
2434599118,2cyclictest4186-21sed00:15:182
2434599118,2cyclictest30739-21chrt19:20:252
2434599118,2cyclictest28776-21ssh00:30:122
2434599118,2cyclictest27976-21cat20:25:012
2434599118,2cyclictest27477-21expr21:35:152
2434599118,2cyclictest26669-21cron19:15:002
2434599118,2cyclictest23635-21grep20:15:132
2434599118,2cyclictest19124-21sh23:25:442
2434599118,2cyclictest11720-21grep19:50:142
2434599118,2cyclictest10530-21sed19:45:242
2434599117,3cyclictest19629-21cat23:45:262
2434299119,2cyclictest821-21expr20:35:151
2434299119,1cyclictest31693-21cat19:25:101
2434299119,1cyclictest20176-21wc20:05:251
2434299119,1cyclictest19538-21ssh21:50:011
2434299119,1cyclictest19538-21ssh21:50:011
2434299119,1cyclictest15324-21chrt21:05:131
2434299119,1cyclictest12029-21taskset20:55:541
2434299118,2cyclictest29064-21idleruntime-cro19:19:591
2434299118,2cyclictest25969-21sed23:30:171
2434299118,2cyclictest2566-21ssh23:55:151
2434299118,2cyclictest23607-21tr20:15:131
2434299110,10cyclictest16553-21sh23:25:061
2434099110,2cyclictest225-21systemd-journal19:39:590
2434899108,2cyclictest20352-21taskset20:05:283
2434899108,2cyclictest13321-21expr21:00:143
2434899108,1cyclictest27561-21cut19:15:153
2434899103,6cyclictest16234-21apt-get20:00:123
2434599108,1cyclictest2106-21expr19:30:142
2434599108,1cyclictest15956-21grep20:00:012
2434599108,1cyclictest15956-21grep20:00:012
2434599107,2cyclictest594-21grep20:35:152
2434299109,1cyclictest27857-21chrt20:24:591
2434299108,1cyclictest47252sleep120:41:211
2434299108,1cyclictest4250-21grep19:35:131
2434299108,1cyclictest29783-21sed20:25:261
2434299106,3cyclictest2417-21grep19:30:161
243459998,1cyclictest7888-21sensors19:40:242
243459997,1cyclictest15490-21expr21:05:132
243429998,1cyclictest29371-21cut19:20:141
243429980,1cyclictest0-21swapper/120:30:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional