You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 13:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Aug 13, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1546629671,9sleep30-21swapper/319:05:363
1549729566,9sleep20-21swapper/219:05:592
1510429371,16sleep00-21swapper/019:05:250
1368128769,14sleep10-21swapper/119:05:101
318062580,0sleep30-21swapper/319:40:223
1593899343,28cyclictest12906-21sh21:30:463
15925993128,2cyclictest94-21kswapd021:37:090
1593899271,7cyclictest8692-21diskmemload00:31:133
1593299271,1cyclictest0-21swapper/100:35:141
1593299263,6cyclictest1537-21ssh22:03:071
1593299260,1cyclictest0-21swapper/122:08:151
1593899259,15cyclictest0-21swapper/323:48:133
1593899250,7cyclictest0-21swapper/322:23:273
1593899250,6cyclictest8692-21diskmemload23:16:473
1593299252,7cyclictest31821-21ssh23:00:131
1593299251,1cyclictest19532-21if_err_enp2s023:50:171
1593299250,2cyclictest0-21swapper/121:49:221
1592599250,1cyclictest0-21swapper/021:50:250
1593899241,2cyclictest0-21swapper/321:23:323
1593899240,5cyclictest0-21swapper/300:04:393
15934992422,1cyclictest0-21swapper/221:40:002
1593299245,18cyclictest8692-21diskmemload21:23:511
1593899232,3cyclictest21619-21grep21:55:233
1593899230,22cyclictest0-21swapper/300:35:143
1593899230,22cyclictest0-21swapper/300:25:143
1593299236,3cyclictest8692-21diskmemload21:10:121
1593299231,22cyclictest0-21swapper/123:38:301
1593299230,4cyclictest0-21swapper/123:10:551
1593299230,22cyclictest0-21swapper/100:05:081
1593899222,6cyclictest8692-21diskmemload22:01:313
1593899220,4cyclictest0-21swapper/323:55:213
15932992219,3cyclictest0-21swapper/121:33:171
1593299220,5cyclictest0-21swapper/100:01:561
1593299220,4cyclictest0-21swapper/121:15:311
1592599220,13cyclictest0-21swapper/021:45:130
1593899212,17cyclictest0-21swapper/321:15:143
1593899210,3cyclictest0-21swapper/323:09:353
1593899210,1cyclictest0-21swapper/323:53:003
1593299211,2cyclictest0-21swapper/100:30:151
1593899201,18cyclictest21901-21tune2fs22:15:133
15938992010,10cyclictest0-21swapper/322:29:423
1593499200,20cyclictest0-21swapper/221:34:572
1593299202,13cyclictest0-21swapper/120:15:131
1593299201,4cyclictest0-21swapper/122:25:151
1593299200,8cyclictest8692-21diskmemload23:07:041
1593299200,5cyclictest0-21swapper/100:27:511
1593299200,4cyclictest0-21swapper/122:15:111
1593899192,7cyclictest11684-21/usr/sbin/munin21:50:173
1593899192,6cyclictest8692-21diskmemload22:44:103
1593899192,4cyclictest20617-21sh19:20:013
1593899190,4cyclictest0-21swapper/323:40:523
1593899190,4cyclictest0-21swapper/322:50:013
1593899190,4cyclictest0-21swapper/322:50:013
1593899190,2cyclictest0-21swapper/323:21:183
1593899190,18cyclictest7242-21cron23:05:003
1593299191,6cyclictest15695-21kworker/1:219:30:261
1593299190,7cyclictest8692-21diskmemload21:42:011
1593299190,19cyclictest0-21swapper/122:33:381
1592599190,18cyclictest0-21swapper/000:10:590
1593899180,3cyclictest0-21swapper/300:05:493
1593899180,17cyclictest0-21swapper/323:32:343
15934991817,1cyclictest0-21swapper/222:35:132
15934991817,0cyclictest0-21swapper/221:45:132
1593299182,4cyclictest16128-21/usr/sbin/munin19:10:111
1593299180,5cyclictest0-21swapper/123:41:261
1592599188,7cyclictest13-21ksoftirqd/023:59:110
201072170,0sleep20-21swapper/223:30:542
1593899179,7cyclictest2351-21sort20:55:173
1593899179,2cyclictest12566-21ssh21:11:313
1593899173,4cyclictest29917-21open_files00:15:223
1593899170,16cyclictest20478-21ssh21:35:253
1593499175,1cyclictest141rcu_preempt00:08:422
1593299179,7cyclictest27017-21ssh23:55:141
1593299179,7cyclictest0-21swapper/122:47:171
1593299179,7cyclictest0-21swapper/122:47:161
1593299174,11cyclictest455-21dbus-daemon19:20:011
1593299171,6cyclictest0-21swapper/120:20:141
1593299171,4cyclictest8724-21ntp_offset20:00:191
1593299170,3cyclictest0-21swapper/100:10:081
1592599173,1cyclictest141rcu_preempt23:02:350
1592599170,1cyclictest0-21swapper/021:41:270
238512160,0sleep10-21swapper/120:30:291
1593899169,6cyclictest2995-21grep19:50:143
1593899169,6cyclictest255662sleep322:36:213
1593899169,2cyclictest22476-21sh22:54:393
1593899169,2cyclictest17475-21sh22:12:273
1593899169,2cyclictest10576-21ssh00:23:393
1593899169,1cyclictest2039-21chrt21:25:113
1593899169,1cyclictest11080-21grep23:25:263
1593899168,2cyclictest27496-21cat20:40:133
1593899168,1cyclictest31813-21ssh23:38:313
1593899161,3cyclictest0-21swapper/322:33:383
1593899161,14cyclictest0-21swapper/319:35:133
1593899160,6cyclictest0-21swapper/300:10:143
1593899160,10cyclictest0-21swapper/322:05:543
15934991614,1cyclictest0-21swapper/221:26:062
1593499160,16cyclictest0-21swapper/223:21:322
1593299169,6cyclictest4779-21ssh23:22:111
1593299169,6cyclictest31811-21ssh00:16:251
1593299169,6cyclictest25375-21grep20:35:181
1593299169,6cyclictest23274-21ssh21:37:211
1593299169,6cyclictest19922-21cut20:25:131
1593299169,6cyclictest19859-21ssh22:52:361
1593299168,2cyclictest13041-21cut22:10:131
1593299162,4cyclictest8692-21diskmemload00:21:101
1593299162,3cyclictest8692-21diskmemload23:29:271
15932991615,1cyclictest38150irq/131-snd_hda23:15:291
15932991613,2cyclictest32136-21expr20:50:131
1593299160,6cyclictest4140-21ssh21:25:301
1593299160,1cyclictest0-21swapper/123:32:331
1592599163,8cyclictest25810-21/usr/sbin/munin23:35:180
1593899159,5cyclictest27113-21ssh21:40:103
1593899159,2cyclictest3115-21ssh21:45:043
1593899159,1cyclictest246112sleep320:35:033
1593899159,1cyclictest11786-21sed20:05:253
1593899158,2cyclictest15217-21cat20:15:153
1593899158,1cyclictest9171-21date20:00:253
1593899158,1cyclictest28961-21ssh22:58:123
1593899158,1cyclictest18074-21head20:20:173
1593899157,2cyclictest14518-21seq20:13:323
1593899152,12cyclictest0-21swapper/319:30:133
1593899151,6cyclictest0-21swapper/319:10:133
1593899151,3cyclictest0-21swapper/323:10:243
15934991514,1cyclictest0-21swapper/223:57:572
1593299159,5cyclictest6111-21ssh22:44:571
1593299159,5cyclictest24451-21uname22:55:181
1593299159,5cyclictest19926-21ssh21:55:031
1593299159,5cyclictest13415-21ssh21:50:251
1593299158,2cyclictest6914-21expr21:05:131
1593299158,1cyclictest10806-21grep23:45:141
1593299157,2cyclictest23763-21ls22:35:201
1592599154,6cyclictest28049-21sendmail21:59:590
15925991510,3cyclictest14007-21ls00:25:200
1593899149,1cyclictest314962sleep320:49:463
1593899147,2cyclictest1633-21chrt19:45:223
1593899147,1cyclictest21709-21cat19:20:153
1593499148,4cyclictest0-21swapper/221:20:222
15934991413,1cyclictest32-21ksoftirqd/221:19:322
15934991413,1cyclictest0-21swapper/200:21:292
1593499141,12cyclictest0-21swapper/222:09:162
15934991410,3cyclictest32080-21grep23:00:142
15934991410,3cyclictest29163-21awk20:45:002
1593299149,4cyclictest30815-21python20:45:241
1593299149,1cyclictest2907-21ntp_states20:55:221
1593299148,1cyclictest32431-21chrt19:43:531
1593299148,1cyclictest29114-21taskset20:44:381
1593299148,1cyclictest23750-21expr19:25:151
1593299148,1cyclictest12679-21grep20:10:131
1593299148,1cyclictest10954-21sh20:05:171
1593299147,2cyclictest6156-21cat19:55:191
1593299147,2cyclictest4943-21cat19:55:001
1593299147,2cyclictest344-21ssh22:21:211
1593299147,1cyclictest4892-21tail21:00:161
1593299141,4cyclictest0-21swapper/119:35:121
1592599149,3cyclictest31354-21wc00:35:250
1592599146,2cyclictest225-21systemd-journal23:15:280
15925991411,2cyclictest8907-21wc23:05:180
15925991411,2cyclictest5558-21wc21:00:240
15925991411,2cyclictest537-21snmpd23:40:300
15925991410,3cyclictest25286-21grep22:55:240
15925991410,3cyclictest19005-21grep20:20:260
15925991410,3cyclictest10877-21grep20:05:160
1592599140,8cyclictest3989-21perf22:05:000
1593899138,1cyclictest11612sleep320:51:063
1593899137,1cyclictest8374-21taskset21:06:103
1593499139,3cyclictest19280-21grep23:50:142
15934991310,2cyclictest31540-21grep22:20:272
15934991310,2cyclictest30559-21grep19:40:142
15934991310,2cyclictest30479-21grep22:00:282
15934991310,2cyclictest25305-21grep23:15:152
15934991310,2cyclictest24070-21ssh23:15:012
15934991310,2cyclictest22087-21cat22:35:002
15934991310,2cyclictest17225-21grep19:10:192
15934991310,2cyclictest1032-21grep20:50:252
1593499130,12cyclictest0-21swapper/221:55:452
1593499130,10cyclictest0-21swapper/222:55:012
1593299137,2cyclictest1132-21sed19:45:181
1592599139,3cyclictest26408-21cut19:30:170
15925991311,1cyclictest1417-21grep23:20:180
15925991310,2cyclictest3951-21grep21:25:270
15925991310,2cyclictest3369-21grep20:55:270
15925991310,2cyclictest21524-21tr00:30:160
15925991310,2cyclictest17814-21tr23:30:140
15925991310,2cyclictest15926-21tail22:50:160
15925991310,2cyclictest13143-21tr22:10:130
1593899129,2cyclictest23340-21hostname19:25:143
1593499129,2cyclictest8532-21cat20:00:182
1593499129,2cyclictest3237-21df19:50:152
1593499129,2cyclictest31852-21cat22:40:192
1593499129,2cyclictest3086-21tr00:00:152
1593499129,2cyclictest21851-21cat00:10:262
1593499129,2cyclictest21146-21grep19:20:122
1593499129,2cyclictest18550-21chrt22:51:342
1593499129,2cyclictest1478-21grep19:45:202
1593499129,2cyclictest11672-21cat20:05:252
1593499122,6cyclictest18213-21latency_hist23:50:002
1593299127,1cyclictest21193-21cstates19:20:121
1592599129,2cyclictest6970-21tr22:45:140
1592599129,2cyclictest6970-21tr22:45:130
1592599129,2cyclictest3688-21sed00:00:170
1592599129,2cyclictest30301-21ssh22:40:010
1592599129,2cyclictest21906-21latency_hist22:34:590
1592599129,2cyclictest21565-21tail20:25:260
1592599129,2cyclictest20400-21grep23:50:250
1592599129,2cyclictest11451-21cut23:45:190
1592599128,2cyclictest29633-21sed19:35:240
1592599128,2cyclictest29263-21wc20:45:000
1592599126,2cyclictest225-21systemd-journal22:45:010
1592599126,2cyclictest225-21systemd-journal22:30:000
1592599125,4cyclictest20628-21munin-run19:20:010
1593899119,2cyclictest21864-21chrt20:28:193
1593899119,1cyclictest4920-21tail21:00:153
1593899118,2cyclictest5078-21seq19:55:013
1593499118,2cyclictest5863-21grep22:25:142
1593499118,2cyclictest32049-21rm00:35:512
1593499118,2cyclictest31673-21ssh21:42:222
1593499118,2cyclictest26301-21sed20:35:262
1593499118,2cyclictest24127-21sshd20:31:512
1593499118,2cyclictest20650-21tr19:20:012
1593499118,2cyclictest19450-21sh00:29:012
1593499118,2cyclictest18662-21ssh21:54:132
1593499118,2cyclictest13173-21grep22:10:132
1593499118,2cyclictest12580-21ssh22:48:262
1593499118,2cyclictest12580-21ssh22:48:252
1593499118,2cyclictest12368-21rm23:26:222
1593499112,8cyclictest28574-21apache_volume00:15:132
1593499111,9cyclictest24732-21ssh00:31:372
1593499110,11cyclictest0-21swapper/223:40:212
1593499110,11cyclictest0-21swapper/222:16:372
1593499110,10cyclictest8692-21diskmemload23:35:432
1593499110,10cyclictest11327-21ssh21:10:322
1592599118,2cyclictest6226-21head19:55:190
1592599118,2cyclictest5010-21ssh22:05:140
1592599118,2cyclictest30096-21wc00:15:240
1592599118,2cyclictest29122-21sh22:20:000
1592599118,2cyclictest2529-21sed19:50:000
1592599118,2cyclictest23411-21tr20:30:230
1592599118,2cyclictest22333-21sh21:17:370
1592599118,2cyclictest17969-21wc23:10:250
1592599118,2cyclictest16148-21grep20:15:210
1592599118,2cyclictest12123-21grep00:05:200
1592599118,2cyclictest1185-21ssh22:21:570
1592599118,2cyclictest11113-21grep21:10:270
1592599112,3cyclictest0-21swapper/023:25:280
1592599111,9cyclictest27830-21/usr/sbin/munin21:20:270
1592599110,3cyclictest225-21systemd-journal21:30:000
1593899109,1cyclictest24220-21taskset20:33:023
1593499108,1cyclictest8249-21expr23:05:152
1593499108,1cyclictest5887-21taskset21:00:282
1593499108,1cyclictest28060-21wget19:35:132
1593499108,1cyclictest26613-21expr19:30:192
1593499108,1cyclictest24347-21cat19:25:202
1593499108,1cyclictest18127-21wc20:20:182
1593499108,1cyclictest12683-21egrep20:10:122
1593499107,2cyclictest30094-21fschecks_time20:45:162
1593499107,2cyclictest20846-21cut20:25:192
1593499107,2cyclictest15312-21cat20:15:152
1593499100,9cyclictest6467-21grep19:55:222
1593499100,9cyclictest3126-21grep20:55:242
1592599108,1cyclictest8725-21sed21:10:000
1592599108,1cyclictest5543-21uname00:20:180
1592599108,1cyclictest30400-21chrt19:40:110
1592599108,1cyclictest16536-21expr19:10:130
1592599108,1cyclictest14524-21chrt20:13:370
1592599107,2cyclictest897-21taskset20:50:240
1592599107,2cyclictest8953-21cut20:00:210
1592599107,2cyclictest3507-21expr19:50:170
1592599107,2cyclictest30040-21cut20:45:160
1592599107,2cyclictest24237-21wc19:25:200
1592599107,2cyclictest21158-21grep19:20:130
159259997,1cyclictest25265-21grep20:35:170
159349986,1cyclictest6768-21tr21:05:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional