You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-22 - 08:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 22, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
726929481,9sleep20-21swapper/219:06:272
736829380,9sleep00-21swapper/019:07:420
725328568,13sleep10-21swapper/119:06:141
751528371,8sleep30-21swapper/319:09:373
165912750,0sleep00-21swapper/022:35:580
74862480,0sleep30-21swapper/323:29:103
763499240,23cyclictest0-21swapper/321:44:153
762499240,1cyclictest0-21swapper/023:41:330
763299210,4cyclictest0-21swapper/222:18:322
763299210,4cyclictest0-21swapper/222:18:322
762899210,19cyclictest0-21swapper/121:17:341
7624992120,1cyclictest0-21swapper/000:23:000
762499211,1cyclictest0-21swapper/022:54:460
763299201,18cyclictest23169-21hwlatdetect22:40:172
763299200,20cyclictest0-21swapper/200:16:462
763299200,18cyclictest0-21swapper/222:29:252
7628992016,3cyclictest2852-21sed23:45:191
762899200,4cyclictest0-21swapper/121:23:111
763299190,19cyclictest0-21swapper/223:59:282
7634991817,1cyclictest0-21swapper/300:29:533
763499180,11cyclictest0-21swapper/320:25:133
763299180,2cyclictest1153-21kworker/u8:200:06:112
762899181,16cyclictest0-21swapper/123:43:451
762899180,18cyclictest0-21swapper/123:24:161
762899180,17cyclictest0-21swapper/100:16:571
7624991810,8cyclictest13-21ksoftirqd/023:34:110
762499180,1cyclictest0-21swapper/021:41:070
762499180,0cyclictest0-21swapper/000:25:440
763499173,6cyclictest225-21systemd-journal20:10:003
763299170,16cyclictest0-21swapper/223:15:342
762899171,1cyclictest25180-21ssh22:21:541
762899170,0cyclictest0-21swapper/121:32:251
762499175,6cyclictest27893-21apt-get22:05:000
762499174,6cyclictest9992-21perf19:15:000
762499170,15cyclictest0-21swapper/021:45:360
7632991613,2cyclictest32-21ksoftirqd/222:54:482
763299160,2cyclictest0-21swapper/200:30:142
763299160,16cyclictest0-21swapper/221:17:222
763299160,14cyclictest0-21swapper/222:31:002
762899160,1cyclictest377-21diskmemload21:51:211
762499168,6cyclictest18279-21sh20:40:010
762499162,3cyclictest0-21swapper/020:40:140
7624991615,1cyclictest0-21swapper/022:29:500
763499158,6cyclictest5921-21grep22:30:183
763299155,3cyclictest141rcu_preempt23:12:502
763299153,1cyclictest28913-21ssh21:45:252
7632991515,0cyclictest0-21swapper/221:23:372
7632991510,2cyclictest0-21swapper/223:34:472
763299150,15cyclictest0-21swapper/223:22:482
762899159,5cyclictest16855-21wc22:55:271
762899154,7cyclictest8788-21awk20:20:001
762899154,7cyclictest8788-21awk20:20:001
762899153,11cyclictest377-21diskmemload22:33:281
762899152,7cyclictest20907-21/usr/sbin/munin00:35:191
7628991515,0cyclictest0-21swapper/121:58:581
7628991514,1cyclictest0-21swapper/100:25:161
7628991510,3cyclictest28185-21rm22:05:001
762499158,6cyclictest9618-21munin-run23:50:000
762499152,9cyclictest8025-21wget23:10:130
7624991511,3cyclictest28194-21grep21:25:250
762499150,2cyclictest0-21swapper/023:29:060
762499150,14cyclictest0-21swapper/022:48:270
763499141,0cyclictest0-21swapper/323:13:473
7634991410,3cyclictest28810-21sed00:00:283
763299149,3cyclictest18551-21ls23:35:192
763299146,4cyclictest225-21systemd-journal20:05:012
7632991413,1cyclictest0-21swapper/223:28:432
7632991410,3cyclictest20701-21sh22:58:212
763299140,2cyclictest0-21swapper/222:06:212
763299140,14cyclictest0-21swapper/221:37:352
762899144,9cyclictest7798-21runrttasks23:29:261
7628991411,2cyclictest14663-21cut22:35:171
7628991410,3cyclictest15216-21wc20:30:261
762899140,4cyclictest0-21swapper/122:40:131
762899140,14cyclictest0-21swapper/123:50:591
762499148,2cyclictest0-21swapper/023:36:220
7624991411,2cyclictest14669-21ssh00:30:290
7624991410,3cyclictest2296-21pluginstate00:05:000
763499139,3cyclictest20863-21cat20:45:003
763499139,3cyclictest15814-21grep22:55:183
763499138,4cyclictest24006-21chrt22:21:033
763499133,1cyclictest225-21systemd-journal20:40:003
7634991310,2cyclictest7618-21ssh21:52:193
7634991310,2cyclictest4290-21grep20:10:153
7634991310,2cyclictest406-21grep23:05:193
7634991310,2cyclictest26732-21sed23:20:273
7634991310,2cyclictest20956-21ls19:35:223
763299139,3cyclictest26809-21wc21:45:002
763299139,2cyclictest25820-21grep20:55:142
7632991310,2cyclictest30423-21grep19:55:182
7632991310,2cyclictest29013-21sh23:03:152
7632991310,2cyclictest2863-21grep21:10:272
7632991310,2cyclictest26514-21grep21:25:142
7632991310,2cyclictest25887-21tr23:40:122
7632991310,2cyclictest25377-21head19:45:182
7632991310,2cyclictest22131-21cat00:35:202
7632991310,2cyclictest11920-21tr00:10:162
762899139,3cyclictest18710-21tr23:55:141
762899138,2cyclictest24449-21tr20:50:191
762899133,9cyclictest27310-21tr21:25:191
762899133,6cyclictest10550irq/122-ahci[0023:20:001
7628991310,2cyclictest76402sleep122:11:181
7628991310,2cyclictest6206-21http20:15:001
7628991310,2cyclictest32605-21sed22:45:271
7628991310,2cyclictest31508-21cut21:05:161
7628991310,2cyclictest24442-21wc23:00:181
7628991310,2cyclictest20457-21sh00:35:001
7628991310,2cyclictest1594-21sed23:05:291
7628991310,2cyclictest15560-21grep19:25:161
762499139,3cyclictest29692-21wc22:05:200
762499139,2cyclictest26203-21grep19:45:260
762499139,2cyclictest21707-21sed20:45:170
762499133,5cyclictest17141-21perf19:30:000
7624991312,1cyclictest0-21swapper/022:15:070
7624991312,1cyclictest0-21swapper/022:15:070
7624991310,2cyclictest9605-21grep21:15:150
7624991310,2cyclictest6753-21grep20:15:150
7624991310,2cyclictest6753-21grep20:15:140
7624991310,2cyclictest32336-21sort20:00:210
7624991310,2cyclictest29781-21tail21:00:280
7624991310,2cyclictest26526-21grep20:55:170
7624991310,2cyclictest21247-21ntpq00:15:220
7624991310,1cyclictest7615-21cyclictest21:57:060
763499129,2cyclictest32491-21sed20:00:203
763499129,2cyclictest29251-21head21:45:263
763499129,2cyclictest27711-21grep23:40:243
763499129,2cyclictest26574-21grep21:25:133
763499129,2cyclictest25312-21head19:45:173
763499129,2cyclictest23548-21ssh22:40:203
763499129,2cyclictest20259-21cut22:00:163
763499129,2cyclictest18140-21head19:30:183
763499129,2cyclictest13788-21cat00:30:213
763499129,2cyclictest12140-21grep21:55:163
763499121,2cyclictest30554-21hddtemp_smartct22:25:173
763499121,10cyclictest9057-21apt-get20:20:143
7634991210,1cyclictest377-21diskmemload00:06:343
7634991210,1cyclictest21795-21tr22:40:003
763299129,2cyclictest6789-21tr20:15:142
763299129,2cyclictest6789-21tr20:15:142
763299129,2cyclictest4857-21grep22:10:162
763299129,2cyclictest29374-21grep21:00:242
763299129,2cyclictest28661-21grep19:50:252
763299129,2cyclictest27073-21cut00:00:152
763299129,2cyclictest2342-21cut21:30:182
763299129,2cyclictest22421-21tr19:40:142
763299129,2cyclictest19326-21sed20:40:172
763299129,2cyclictest1869-21basename23:45:142
763299129,2cyclictest12025-21sort20:25:162
763299120,12cyclictest0-21swapper/223:53:012
763299120,12cyclictest0-21swapper/222:00:542
762899129,2cyclictest9373-21tr23:30:151
762899129,2cyclictest28863-21head00:00:281
762899129,2cyclictest21279-21tr22:00:211
762899129,2cyclictest20747-21readlink22:20:011
762899129,2cyclictest20747-21readlink22:20:001
762899129,2cyclictest19700-21grep21:40:161
762899129,2cyclictest18699-21ssh23:35:201
762899129,2cyclictest11748-21nfs4_client21:35:191
762899128,3cyclictest9971-21turbostat19:14:591
762899124,4cyclictest38150irq/131-snd_hda19:38:161
762899122,2cyclictest377-21diskmemload23:12:291
7628991210,1cyclictest377-21diskmemload22:53:171
762899120,11cyclictest2837-21sed21:10:271
762899120,11cyclictest23763-21grep19:40:261
762499129,2cyclictest6439-21grep22:30:200
762499129,2cyclictest4321-21tr20:10:140
762499129,2cyclictest31614-21wc20:00:000
762499129,2cyclictest27744-21grep19:50:170
762499129,2cyclictest276772sleep022:43:030
762499129,2cyclictest22913-21cut19:40:170
762499129,2cyclictest21240-21grep00:35:150
762499129,2cyclictest20495-21grep19:35:190
762499129,2cyclictest15815-21grep22:55:180
762499129,2cyclictest1571-21grep23:05:290
762499129,2cyclictest13863-21tr19:20:200
762499129,2cyclictest13506-21sed00:10:280
762499129,2cyclictest12040-21sed23:50:280
762499129,2cyclictest11852-21grep19:15:260
762499129,2cyclictest10769-21head20:20:260
762499128,2cyclictest2447-21cut20:05:180
762499128,2cyclictest17424-21cut19:30:150
762499128,2cyclictest10555-21cut21:35:120
762499121,5cyclictest225-21systemd-journal21:55:000
762499120,9cyclictest3272-21tr00:05:150
245822120,2sleep0161rcuc/023:00:180
763499119,1cyclictest20301-21grep23:55:253
763499118,2cyclictest9926-21ssh21:15:163
763499118,2cyclictest9755-21ssh22:51:173
763499118,2cyclictest8901-21ssh23:30:063
763499118,2cyclictest871-21ssh22:07:323
763499118,2cyclictest5927-21rm23:47:093
763499118,2cyclictest5424-21ssh21:31:263
763499118,2cyclictest4499-21rm22:10:123
763499118,2cyclictest4348-21rm21:11:323
763499118,2cyclictest31023-21tr22:45:163
763499118,2cyclictest30642-21ssh00:20:413
763499118,2cyclictest27714-21ssh00:39:093
763499118,2cyclictest26529-21sh23:01:183
763499118,2cyclictest22918-21ssh00:16:123
763499118,2cyclictest18456-21ssh23:15:323
763499118,2cyclictest18215-21grep23:35:183
763499118,2cyclictest17611-21ssh21:20:023
763499118,2cyclictest15806-21ssh23:53:203
763499118,2cyclictest15275-21ssh00:11:463
763499118,2cyclictest14798-21sed22:15:273
763499118,2cyclictest14798-21sed22:15:273
763499118,2cyclictest11113-21if_enp2s019:15:183
763499117,3cyclictest8135-21tr19:10:163
763499117,2cyclictest31923-21ls21:05:203
763499117,2cyclictest13898-21wc19:20:203
763499116,2cyclictest10478-21seq21:35:123
763299118,2cyclictest6308-21ssh00:25:292
763299118,2cyclictest4308-21grep20:10:142
763299118,2cyclictest26476-21ssh22:22:552
763299118,2cyclictest24511-21ls20:50:212
763299118,2cyclictest2271-21ssh00:23:512
763299118,2cyclictest21195-21grep20:45:162
763299118,2cyclictest1610-21ssh22:46:282
763299118,2cyclictest1608-21sed23:05:292
763299118,2cyclictest14785-21sed22:35:162
763299118,2cyclictest10126-21sed20:20:202
763299117,3cyclictest8353-21df19:10:172
763299111,9cyclictest3198-21apache_processe21:50:122
763299110,0cyclictest0-21swapper/221:58:392
762899118,2cyclictest356-21grep20:00:241
762899118,2cyclictest31805-21ssh21:47:241
762899118,2cyclictest30270-21grep00:20:271
762899118,2cyclictest22580-21grep20:45:261
762899118,2cyclictest1184-21ssh22:26:581
762899118,2cyclictest11449-21ssh00:10:081
762899117,3cyclictest38150irq/131-snd_hda00:05:151
762899117,2cyclictest16461-21grep20:35:161
762899117,1cyclictest26361-21taskset19:45:481
762899114,3cyclictest225-21systemd-journal19:30:001
762899111,4cyclictest225-21systemd-journal20:00:001
762499119,1cyclictest24788-21cut23:20:140
762499118,2cyclictest21947-21ssh23:18:160
762499118,2cyclictest20359-21ssh21:20:500
762499118,2cyclictest18452-21ssh23:55:130
762499118,2cyclictest14476-21cut20:30:170
762499112,8cyclictest2210-21/usr/sbin/munin21:30:160
7624991111,0cyclictest0-21swapper/021:11:360
763499108,1cyclictest275382sleep320:56:033
763499108,1cyclictest27152-21awk19:50:133
763499108,1cyclictest23541-21expr20:50:143
763499108,1cyclictest16707-21grep19:25:273
763499108,1cyclictest13805-21apt-get20:30:143
763499107,2cyclictest22806-21taskset20:47:013
763499107,2cyclictest22358-21cat19:40:143
763299108,1cyclictest3462sleep221:09:482
763299108,1cyclictest20056-21cstates19:35:162
763299108,1cyclictest18086-21seq20:37:372
763299108,1cyclictest14655-21if_enp0s31f620:30:192
763299107,2cyclictest2764-21cat20:05:212
763299107,2cyclictest19050-21grep19:30:272
763299107,2cyclictest16600-21cat19:25:262
763299107,2cyclictest13413-21grep19:20:172
763299107,2cyclictest10965-21grep19:15:172
762899108,1cyclictest9346-21cat20:20:151
762899108,1cyclictest29894-21taskset21:00:331
762899108,1cyclictest20633-21chrt20:44:351
762899108,1cyclictest12205-21taskset19:18:151
762899107,2cyclictest3868-21wc20:10:001
762899100,9cyclictest27599-21chrt20:56:491
762899100,9cyclictest27523-21sed19:50:141
762499108,1cyclictest22371-21ssh22:20:170
762499108,1cyclictest10309-21ssh22:13:240
762499107,2cyclictest24306-21wc20:50:180
762499107,2cyclictest11696-21tr20:25:140
762499104,5cyclictest32413-21unixbench_singl21:05:250
61592108,1sleep36576-21cat20:15:013
61592108,1sleep36576-21cat20:15:013
76349997,1cyclictest31238-21sshd19:57:283
76349997,1cyclictest29017-21if_enp2s021:00:193
76289997,1cyclictest12930-21wc20:25:251
76289996,2cyclictest14486-21taskset19:21:491
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional