You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 08, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27863210882,19sleep20-21swapper/207:05:242
28708210375,9sleep10-21swapper/107:09:041
2847429379,10sleep30-21swapper/307:06:013
2846629065,9sleep00-21swapper/007:05:540
2887299274,20cyclictest0-21swapper/307:30:013
2887299272,22cyclictest6286-21idleruntime-cro10:20:003
2887299271,24cyclictest0-21swapper/307:35:143
2887299263,22cyclictest21967-21diskmemload11:21:253
2887299261,12cyclictest0-21swapper/310:30:033
2887299257,15cyclictest21967-21diskmemload11:48:343
2887299252,20cyclictest27992-21grep10:10:173
2887299250,1cyclictest0-21swapper/310:29:583
2886599250,14cyclictest0-21swapper/107:50:151
2887299243,20cyclictest14766-21munin-run08:55:003
2887299242,21cyclictest21967-21diskmemload12:09:413
2887299240,23cyclictest0-21swapper/311:58:273
2887299233,19cyclictest21967-21diskmemload12:01:133
2887299231,19cyclictest21967-21diskmemload09:32:193
2887299230,1cyclictest21967-21diskmemload11:28:463
2887299230,1cyclictest0-21swapper/310:04:453
2886899230,22cyclictest0-21swapper/210:30:012
2887299222,19cyclictest21967-21diskmemload09:15:303
2887299222,18cyclictest21967-21diskmemload12:36:423
28872992211,10cyclictest7744-21snmpd11:50:133
2887299220,2cyclictest8901-21taskset10:20:493
2887299220,19cyclictest0-21swapper/311:15:213
2886599220,1cyclictest0-21swapper/111:42:341
2887299213,17cyclictest21967-21diskmemload10:42:223
2887299213,15cyclictest21967-21diskmemload09:51:373
2887299212,18cyclictest25226-21taskset11:00:263
28872992117,4cyclictest0-21swapper/310:45:323
2887299211,1cyclictest21967-21diskmemload12:25:263
2887299210,2cyclictest3289-21unixbench_singl09:20:353
2887299210,2cyclictest0-21swapper/311:31:163
2887299210,2cyclictest0-21swapper/310:59:233
2887299210,20cyclictest0-21swapper/309:11:123
2887299210,20cyclictest0-21swapper/307:10:263
28868992110,10cyclictest13003-21kworker/2:008:12:102
2886899210,21cyclictest0-21swapper/212:28:292
2887299202,18cyclictest21967-21diskmemload12:14:463
2887299200,1cyclictest21967-21diskmemload09:37:383
2887299200,19cyclictest0-21swapper/307:55:223
2886899200,2cyclictest0-21swapper/210:48:262
2886899200,1cyclictest0-21swapper/207:40:202
2886899200,19cyclictest0-21swapper/212:05:172
2887299194,14cyclictest31507-21sh10:40:003
2887299192,17cyclictest21967-21diskmemload09:40:423
2887299191,17cyclictest26339-21grep08:10:153
2887299190,2cyclictest0-21swapper/310:05:133
2887299190,1cyclictest0-21swapper/312:34:233
2887299190,1cyclictest0-21swapper/310:53:103
2887299190,1cyclictest0-21swapper/310:53:093
2886899192,1cyclictest0-21swapper/212:33:282
2886899190,3cyclictest0-21swapper/209:49:492
2886899190,15cyclictest0-21swapper/210:37:072
2886199190,18cyclictest0-21swapper/010:05:160
2887299187,8cyclictest27688-21ssh12:21:513
2887299181,1cyclictest17341-21apt-get12:15:143
2887299181,16cyclictest21967-21diskmemload11:10:083
2887299181,16cyclictest0-21swapper/311:38:203
2887299180,17cyclictest0-21swapper/311:42:523
28868991816,2cyclictest9997-21kworker/2:010:24:292
2886899180,16cyclictest0-21swapper/211:01:592
2886199180,2cyclictest0-21swapper/011:08:360
2887299173,13cyclictest0-21swapper/308:55:153
2887299173,13cyclictest0-21swapper/308:55:153
2887299172,14cyclictest2375-21/usr/sbin/munin07:20:183
2887299172,12cyclictest5878-21entropy08:35:243
2887299171,1cyclictest0-21swapper/309:27:243
2887299171,14cyclictest21967-21diskmemload09:45:533
2887299170,1cyclictest0-21swapper/308:45:003
2887299170,16cyclictest0-21swapper/311:05:323
28868991716,1cyclictest0-21swapper/210:54:562
28868991716,1cyclictest0-21swapper/210:54:552
2886199176,7cyclictest0-21swapper/009:25:180
2886199174,7cyclictest19463-21awk12:40:010
2887299161,4cyclictest2485-21gpgv08:30:003
2887299160,15cyclictest0-21swapper/309:58:343
28868991611,3cyclictest8904-21munin-run09:55:002
2886899160,2cyclictest30791-21sensors_temp11:05:242
2886899160,1cyclictest0-21swapper/211:57:442
28865991612,3cyclictest12451-21ls07:40:271
28865991611,2cyclictest25-21ksoftirqd/110:27:241
2886599160,16cyclictest0-21swapper/111:45:571
2886199165,6cyclictest30107-21awk08:20:000
2886199165,6cyclictest2442-21munin-run08:30:000
2887299159,5cyclictest21320-21grep08:00:163
2887299158,6cyclictest13315-21date07:44:593
2887299150,11cyclictest0-21swapper/307:30:133
2886899155,5cyclictest224-21systemd-journal08:04:592
2886899153,10cyclictest0-21swapper/212:17:382
2886899152,5cyclictest10744-21idleruntime-cro07:40:012
28868991514,1cyclictest23993-21kworker/2:211:31:292
28868991510,3cyclictest0-21swapper/210:13:572
2886899150,0cyclictest0-21swapper/211:25:472
2886599150,14cyclictest0-21swapper/111:25:441
2887299148,5cyclictest23533-21expr08:05:173
2887299148,5cyclictest21211-21which09:05:233
2887299148,5cyclictest17865-21seq07:54:533
2887299148,5cyclictest13708-21cat07:45:163
2887299148,5cyclictest12074-21seq08:46:113
2887299147,6cyclictest30142-21cat08:20:003
2887299141,1cyclictest17465-21basename09:00:163
2887299141,12cyclictest4143-21grep08:30:243
2886899149,4cyclictest17410-21taskset10:00:422
2886899141,8cyclictest1-21systemd07:20:012
2886899140,7cyclictest0-21swapper/211:55:002
2886599149,4cyclictest0-21swapper/112:20:171
28865991413,1cyclictest0-21swapper/112:03:451
28865991411,2cyclictest4439-21grep11:10:281
28865991410,2cyclictest5551-21tr08:35:221
28861991410,3cyclictest19145-21grep12:15:260
28861991410,2cyclictest772-21sed07:15:260
2887299136,6cyclictest30796-21expr08:20:163
2887299131,3cyclictest938-21taskset07:17:153
2886899139,3cyclictest23977-21grep09:40:172
2886899137,5cyclictest25371-21cut12:20:232
2886899133,2cyclictest224-21systemd-journal11:45:002
2886899132,6cyclictest141rcu_preempt12:40:012
28868991312,1cyclictest24851-21kworker/2:109:15:292
28868991312,1cyclictest0-21swapper/208:23:102
28868991310,2cyclictest20626-21cut09:05:182
28868991310,2cyclictest18849-21cat10:30:012
2886899130,2cyclictest0-21swapper/207:20:122
2886599139,3cyclictest23579-21tail09:10:191
2886599139,3cyclictest1759-21wc08:25:231
2886599133,1cyclictest141rcu_preempt10:42:531
28865991310,2cyclictest6268-21cron10:19:591
28865991310,2cyclictest4121-21grep07:25:161
28865991310,2cyclictest31012-21grep08:20:181
28865991310,2cyclictest29920-21wc09:15:281
28865991310,2cyclictest27290-21ssh10:35:241
28865991310,2cyclictest22261-21grep10:05:191
28865991310,2cyclictest14320-21tail09:30:301
28865991310,2cyclictest1409-21ssh11:35:191
2886599130,13cyclictest0-21swapper/112:36:081
2886599130,12cyclictest7853-21grep07:30:261
2886599130,12cyclictest2685-21cron09:50:001
2886199139,2cyclictest6229-21cat10:45:180
2886199136,6cyclictest17895-21munin-run07:55:000
2886199133,5cyclictest0-21swapper/008:54:590
2886199132,3cyclictest0-21swapper/010:50:140
2886199132,3cyclictest0-21swapper/010:50:140
2886199131,7cyclictest25322-21munin-run08:10:000
28861991311,1cyclictest0-21swapper/011:30:140
28861991310,2cyclictest512-21grep12:25:260
28861991310,2cyclictest4929-21tr07:25:210
28861991310,2cyclictest4211-21tr12:05:190
28861991310,2cyclictest21334-21cat09:05:260
28861991310,2cyclictest11094-21grep07:40:140
2886899129,2cyclictest9593-21strings11:15:192
2886899129,2cyclictest8355-21grep08:40:172
2886899129,2cyclictest5939-21cut08:35:242
2886899129,2cyclictest24200-21tail09:10:262
2886899129,2cyclictest13841-21ls08:50:232
2886899129,2cyclictest10012-21ssh12:10:122
2886899128,3cyclictest9913-21ssh09:27:092
2886899128,3cyclictest31325-21cat07:15:012
2886899128,3cyclictest22312-21head10:05:192
2886899128,3cyclictest19546-21perf09:05:002
2886899128,3cyclictest10505-21tr08:45:192
2886899125,6cyclictest13189-21/usr/sbin/munin11:45:162
2886899123,8cyclictest17993-21apt-get10:55:142
2886899123,5cyclictest5029-21apt08:34:592
2886899121,11cyclictest16462-21kworker/2:209:35:152
28868991210,1cyclictest14293-21grep09:30:302
2886599129,2cyclictest7432-21wc10:45:291
2886599129,2cyclictest27375-21grep11:30:181
2886599129,2cyclictest25531-21grep11:55:201
2886599129,2cyclictest24159-21grep09:40:191
2886599129,2cyclictest22714-21cron08:05:001
2886599129,2cyclictest2265-21tail09:20:231
2886599129,2cyclictest19515-21cut11:50:191
2886599129,2cyclictest18736-21tr09:35:181
2886599129,2cyclictest11862-21grep12:10:261
2886599128,3cyclictest9783-21cron08:44:591
2886599128,3cyclictest30049-21sed11:05:171
2886599120,11cyclictest8568-21tail10:20:281
2886599120,11cyclictest15373-21cut10:00:151
2886199129,2cyclictest8785-21grep08:40:190
2886199129,2cyclictest6532-21sort12:30:180
2886199129,2cyclictest6016-21ssh10:19:430
2886199129,2cyclictest3897-21grep09:50:160
2886199129,2cyclictest31544-21grep10:39:590
2886199129,2cyclictest29700-21grep07:10:170
2886199129,2cyclictest28927-21perf11:05:010
2886199129,2cyclictest2831-21rm09:20:310
2886199129,2cyclictest27270-21sed08:10:260
2886199129,2cyclictest26571-21grep11:55:290
2886199129,2cyclictest2621-21tr11:10:140
2886199129,2cyclictest14273-21grep09:30:300
2886199129,2cyclictest13680-21head10:25:200
2886199129,2cyclictest11326-21sed12:10:220
2886199129,2cyclictest10240-21sed07:35:270
2886199128,3cyclictest19445-21cut10:30:160
2886199126,5cyclictest26175-21munin-run11:30:000
2886199126,2cyclictest224-21systemd-journal11:40:000
2886199124,5cyclictest29420-21ssh09:45:140
2886199120,6cyclictest7425-21basename08:40:000
2886899118,2cyclictest6862-21cut07:30:162
2886899118,2cyclictest32629-21ssh11:35:002
2886899118,2cyclictest31916-21sh10:40:102
2886899118,2cyclictest30067-21cron08:20:002
2886899118,2cyclictest2510-21ssh12:03:442
2886899118,2cyclictest23827-21cut08:05:182
2886899118,2cyclictest2258-21grep09:20:232
2886899118,2cyclictest15697-21ssh11:20:222
2886899118,2cyclictest14501-21grep07:45:202
2886899117,3cyclictest16502-21cat08:55:272
2886899117,3cyclictest16502-21cat08:55:272
2886899117,3cyclictest1624-21tr10:15:172
2886899110,0cyclictest0-21swapper/209:56:552
2886599118,2cyclictest9733-21grep07:35:211
2886599118,2cyclictest8430-21tail09:25:281
2886599118,2cyclictest763-21head07:15:261
2886599118,2cyclictest7001-21ssh12:30:201
2886599118,2cyclictest5244-21ssh09:50:391
2886599118,2cyclictest30168-21sort08:20:001
2886599118,2cyclictest28959-21ssh10:10:231
2886599118,2cyclictest25166-21chrt08:08:181
2886599118,2cyclictest21500-21grep09:05:271
2886599118,2cyclictest17491-21tr09:00:171
2886599118,2cyclictest17386-21grep12:15:131
2886599118,2cyclictest17211-21sh10:54:361
2886599118,2cyclictest17211-21sh10:54:361
2886599118,2cyclictest15062-21sed11:20:161
2886599118,2cyclictest11932-21sh09:56:301
2886599118,2cyclictest10292-21sed11:15:261
2886599117,3cyclictest5338-21wc12:05:261
2886599114,4cyclictest5024-21awk08:34:591
2886599110,11cyclictest0-21swapper/112:25:501
2886599110,10cyclictest18947-21cat10:55:191
2886199118,2cyclictest7280-21tail11:40:190
2886199118,2cyclictest4806-21sh10:44:200
2886199118,2cyclictest3445-21cat12:05:000
2886199118,2cyclictest31695-21rm09:17:580
2886199118,2cyclictest30512-21wc08:20:160
2886199118,2cyclictest25007-21uniq12:20:200
2886199118,2cyclictest23521-21expr09:10:200
2886199118,2cyclictest23289-21ssh09:40:090
2886199118,2cyclictest20857-21cut08:00:140
2886199118,2cyclictest19953-21cat11:50:210
2886199118,2cyclictest19048-21cat10:55:210
2886199118,2cyclictest18188-21grep09:00:200
2886199118,2cyclictest16065-21rm11:47:200
2886199118,2cyclictest15256-21grep11:20:180
2886199118,2cyclictest12883-21sh09:57:340
2886199118,2cyclictest12220-21ssh10:24:440
2886199118,2cyclictest11050-21ssh11:16:240
2886199117,3cyclictest7535-21find07:30:230
2886199112,7cyclictest19047-21df_inode09:35:190
2886199110,10cyclictest19789-21ssh10:03:240
2886899108,2cyclictest17394-21grep07:50:262
2886899108,1cyclictest2532-21cat08:30:002
2886899108,1cyclictest20036-21taskset07:56:282
2886899106,3cyclictest2591-21ssh11:10:142
2886899100,9cyclictest37850irq/131-snd_hda07:25:142
2886599108,1cyclictest145622sleep107:45:221
2886599108,1cyclictest14466-21chrt08:51:241
2886599107,2cyclictest29240-21cut07:10:141
2886599107,2cyclictest27551-21taskset08:13:211
2886599107,2cyclictest23847-21ssh11:00:081
2886599107,2cyclictest20234-21cut10:30:201
2886599107,2cyclictest15769-21grep08:55:191
2886599107,2cyclictest15769-21grep08:55:191
2886599107,2cyclictest10506-21tr08:45:191
2886199108,1cyclictest13835-21cat07:45:160
2886199107,2cyclictest3552-21tune2fs08:30:190
2886199107,2cyclictest28508-21tail10:10:190
2886199107,2cyclictest10227-21chrt08:45:030
2886199102,4cyclictest224-21systemd-journal09:00:000
2886199102,4cyclictest224-21systemd-journal08:59:590
2886199100,9cyclictest19536-21cut07:55:250
288659997,1cyclictest19516-21which07:55:251
288659996,2cyclictest1727-21tr07:20:131
288619997,1cyclictest1672-21cat07:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional